Algorithm

A common need in computer vision is to obtain good vector representations (embeddings) of images that can be used for downstream tasks such as clustering, classification, detection, segmentation, etc. For example, a simple classification task may consist of the following steps:

1. Calculating representations of a set of images
2. Storing and indexing them in a vector database
3. Determining the class using a simple algorithm like k-NN or a linear model

It's not always possible to build a model to compute good representations for a given problem due to the availability of training data: annotated images may not be available, expensive to produce, or nonexistent at the beginning of a project. Therefore, it is interesting to consider generic models pre-trained on large, general-purpose datasets. For a long time, our reference models at Sicara were CNNs (typically and in chronological order: VGG, ResNet, EfficientNet) pre-trained on ImageNet-1K (~1.2M images).

Over the past 5 years, several innovations have changed the landscape:

• Advancements in self-supervised training techniques, which eliminate the need for annotation during training and therefore allow leveraging much larger image datasets (e.g., 1.2B images for DINO V2).
• The emergence of transformers, a new architecture based on attention mechanisms (first in NLP with BERT and then in vision with ViT) that surpass CNNs in terms of performance, flexibility, and scalability to large datasets. Meta's DINO (2021) combines and benefits from both approaches.
• Meta has made the pre-trained model weights public, offering several advantages such as commercial usage under the Apache 2.0 license, availability on torch hub, various model sizes ranging from 21M to 1.1B parameters, and the ability to construct highly performing classifiers without fine-tuning.

OUR PERSPECTIVE

We recommend using the DINO V2 models (architecture and weights) as off-the-shelf embedding models without retraining. Fine-tuning or full training with DINO V2 is recommended for experienced users with access to high computational power and large volumes of data.

GPT-4, the most powerful Large Language Model (LLM) currently developed by OpenAI, is an autoregressive text generation model. It is likely composed of a "Mixture of Experts" of several models, each with over 200 billion parameters, as confirmed by several leaks. GPT-4 can also accept images as input and can handle most text or image processing tasks such as translation, synthesis, comprehension, and writing, as demonstrated by using ChatGPT Plus. GPT-4 finds its utility in various domains such as information retrieval, automation of customer support workflows, and automated document processing. In nearly all tasks, GPT-4 significantly outperforms open-source alternatives and clearly surpasses other proprietary models (Claude, PalM, etc.).

The Function Calling feature integrated into GPT-4 allows the use of predefined “functions” or templates in model calls. This facilitates the generation of structured outputs, rather than being limited to raw text. Functions also enable interaction with external tools such as executing code, accessing APIs, or running shell commands.

Recently, OpenAI released GPT-4 Turbo, which is an iteration of GPT-4 with approximately 3x lower costs, faster inference time, and a maximum context window of 128k tokens (40k words). However, it remains relatively slow for inference compared to models like GPT-3.5, which is more suitable for real time applications, and it is expensive ($0.1 for a call with 4k tokens input and 2k tokens output). Finally, be cautious of the reliability of the Turbo version, which is still in beta and may have less reliable response accuracy. In contexts where output quality is particularly critical, we recommend sticking to the stable version of GPT-4 for now.

OUR PERSPECTIVE

GPT-4 surpasses all publicly accessible models and is particularly resilient to hallucinations. The integration of Function Calling and OpenAI Assistants API, along with the recent cost reduction with GPT-4 Turbo, further strengthens its position as a leader.

If reducing cost or latency is more important than raw performance, then prefer GPT-3.5 or another "smaller" LLM. For specific security or control constraints, there are also open-source LLMs available.

The widespread adoption of Machine Learning in the 2010s owes much to the advent of open-source libraries designed for tensor and gradient calculations, notably PyTorch in 2016 and TensorFlow in 2015.

PyTorch, a brainchild of MetaAI initially focused on research, has broadened its reach to industry applications. It facilitates the setup and training of Machine Learning models, including neural networks, and supports the crucial data preprocessing needed for model training. PyTorch is celebrated for its balance between detailed control over training loops and user-friendly syntax. Additions like PyTorch Lightning have further eased the syntax and streamlined deployment processes.

A standout feature of PyTorch is its extensive model library, courtesy of the active community. Platforms like Papers With Code and HuggingFace showcase a predominance of PyTorch-based models over TensorFlow. PyTorch also shines in backward compatibility, offering a safer environment for updates—a critical aspect for production deployments.

OUR PERSPECTIVE

We strongly advocate for PyTorch, praising its versatility and the vibrant community supporting it. However, for projects involving embedded AI, caution is advised as PyTorch Mobile is still under development. Despite this, PyTorch stands as a robust option for a broad range of applications, providing a well-rounded toolkit for developing and training Machine Learning models.

Large Language Models (LLMs) have risen to prominence for their ability to perform zero-shot tasks and serve as a dynamic universal knowledge repository. However, their knowledge is confined to publicly available data up until their last training update.

Retrieval Augmented Generation (RAG) enhances LLMs by integrating them with an external knowledge base, such as Notion, Confluence, PDFs, and internal documents. This integration allows for natural language queries directly to these databases.

The RAG approach operates in two phases:

1. Retrieval: Identifies the documents most relevant to a query by comparing the query's vector embedding with document representations in a database. This can be done using standard or dedicated vector databases (see Vector Search with Standard Databases and Dedicated Vector Databases).
2. Generation: Leverages an LLM to formulate responses based on the information retrieved in the first step.

This methodology offers several benefits, including the avoidance of high GPU costs associated with LLMs, improved explainability through traceable data sources for answers, the ability to stay current with updates to the knowledge base without retraining, enhanced security by restricting sensitive data access, and reduced instances of model-generated inaccuracies by anchoring responses in sourced documents.

However, RAG setups are more complex and potentially costlier than using a singular model, especially if one opts for supervised LLM fine-tuning. The absence of fine-tuning could also limit the system's understanding of domain-specific terminology and the initial retrieval step may be a precision bottleneck—failing to source relevant information compromises the LLM's response accuracy.

OUR PERSPECTIVE

Despite these challenges, RAG systems are often favored over direct fine-tuning methods when it's crucial to manage the data underpinning generated responses, such as for security reasons or when dealing with frequently updated information (like daily document revisions).

Implementing an initial version of a RAG system to explore its potential is quite feasible without extensive technical expertise, thanks to tools like ChatGPT+ GPTs, OpenAI's Assistants API, and frameworks like LangChain. Nevertheless, optimizing the system to fully leverage its capabilities typically necessitates further customization and iteration.

Complex machine learning models are often viewed as inscrutable "black boxes" due to their low interpretability. To build trust in these models, various interpretability methods have been developed. SHAP (SHapley Additive exPlanations) stands out as a method that demystifies the decision-making process of any model, making it particularly valuable for models using structured data.

Before SHAP, interpretation of complex ML models largely depended on simpler approaches like feature importance. SHAP differentiates itself by employing cooperative game theory to assess the contribution of each feature to the model's prediction on a local level.

One of SHAP's strengths is its local focus, enabling detailed examination of how each variable affects individual predictions or those within a specific dataset cluster. Unlike basic interpretive methods that might only offer a generalized view, SHAP values can be aggregated and visualized, providing a deeper insight into the model's workings beyond what a simple bar chart could convey.

However, SHAP's major drawback is its computational demand, especially with models that have numerous parameters or when applied to extensive datasets.

OUR PERSPECTIVE

In our projects, especially in sectors where understanding model decisions is critical—such as finance and healthcare—we consistently employ SHAP for its precise interpretative power with structured data. It not only aids in ensuring compliance and transparency but also helps in refining our models through a meticulous analysis of input contributions.

Given its benefits, we advocate for the adoption of SHAP, particularly in scenarios where detailed model interpretability is essential.

SpaCy stands out as a production-oriented open-source Python library for natural language processing (NLP), prized for its efficiency and focus on delivering "black box" models for swift deployment and dependable outcomes. In contrast to research-focused NLP tools like NLTK or Stanford NLP, SpaCy aims to offer practical, high-performance solutions across a broad spectrum of NLP tasks, including named entity recognition and text classification.

One of SpaCy's significant advantages is its simplicity. The library facilitates rapid model training through configuration files and integrates command-line interface (CLI) commands as Python functions, streamlining SpaCy's incorporation into various development workflows. Its object-oriented design enhances usability, making it accessible even to those relatively new to NLP. Additionally, SpaCy supports a comprehensive range of models from the HuggingFace Transformers library and multiple proprietary and open-source Large Language Models (LLMs) through APIs like OpenAI and HuggingFace, whether fine-tuned or not.

However, it's worth noting that while SpaCy allows for iterative training data and hyperparameter adjustments, it offers limited flexibility for deep modifications to the underlying models. This design choice reflects SpaCy's commitment to achieving quick, reliable results and simplifying the development process.

OUR PERSPECTIVE

SpaCy has established itself as a dependable tool in the NLP domain, making it a strong candidate for traditional NLP projects where robustness and performance are key considerations.

As we haven't yet utilized SpaCy with LLMs in a production environment, we refrain from offering guidance on its application in such contexts.

The YOLO (You Only Look Once) series represents a groundbreaking approach in the realm of real-time object detection algorithms. Debuting in 2016, the inaugural YOLO model set itself apart from predecessors like R-CNN and Fast R-CNN by its remarkable inference speed. This leap in performance is largely attributed to YOLO's unique architecture, which merges the region proposal and classification steps into a single pass through the neural network, hence its name.

YOLO has seen multiple iterations, each improving upon the last. Notably, YOLOv4, introduced in 2020, achieved significant advancements over its contemporaries, such as Faster RCNN-FPN+, boasting a fivefold increase in speed and a 9.4-point improvement in Box AP on the MS COCO benchmark, a standard for measuring object detection precision.

Following YOLOv4, Ultralytics assumed development, releasing subsequent versions under the AGPL-3.0 license, complicating their commercial application. Despite only modest performance gains, these iterations are praised for their high code quality, enhancing usability. The most recent YOLO versions continue to lead in real-time object detection benchmarks and have expanded their application to tasks like segmentation, object tracking, and pose estimation.

OUR PERSPECTIVE

For projects demanding swift object detection, YOLO models are a solid choice, offering both speed and precision. However, alternatives that trade off some speed for increased accuracy exist.

Our experience primarily involves YOLOv4, which has proven effective in various settings, including on embedded systems through tflite compression. Although the open-source nature of these models has occasionally presented challenges in training, these hurdles were surmountable. Due to licensing constraints, we advise against using Ultralytics's iterations (v5 and onwards).