arduino-extlib

The arduino-extlib.py is a Python utility designed to manage external library submodules in Arduino library projects. It provides a flexible way to integrate external dependencies while supporting both development (symlink) and distribution (copy) workflows.

Overview

This tool addresses the common challenge of integrating external C/C++ libraries into Arduino projects. It allows developers to:

  • Work with live submodule updates during development using symbolic links

  • Create stable copies for distribution and release builds

  • Verify that copied files match their source submodule versions

  • Maintain version control and integrity of external dependencies

The tool operates based on a .extlib.yml configuration file that defines how external libraries should be integrated into your Arduino project.

Motivation

Arduino libraries often depend on external C/C++ libraries that are maintained as separate repositories. Managing these dependencies presents several challenges:

Development Workflow

During active development, you want changes in external libraries to be immediately reflected in your project. Traditional copying approaches require manual synchronization.

Distribution Workflow

For releases and distribution, you need stable, versioned copies of external code that won’t change unexpectedly. Symbolic links are not suitable for distribution packages.

Version Control

You need to track which exact version of each external library is being used, including detecting uncommitted changes.

File Management

Different projects may need different subsets of files from external libraries, and the destination structure may differ from the source.

The arduino-extlib.py tool solves these challenges by providing:

  • Dual-mode operation: Seamless switching between symlink (development) and copy (distribution) modes

  • Automatic version tracking: Git tags and commit hashes are automatically recorded

  • Flexible file mapping: Configure exactly which files go where

  • Integrity verification: Ensure copied files haven’t been modified locally

  • Dirty state detection: Detect uncommitted changes in submodules

Installation and Setup

Prerequisites

  • Python 3.6 or higher

  • Git command line tools

  • PyYAML library

Install PyYAML if not already available:

pip install PyYAML

Usage

The tool provides three main commands:

Command Line Interface

python arduino-extlib.py [OPTIONS] COMMAND [COMMAND_OPTIONS]

Global Options

-r, --root-path PATH

Path to the Arduino library root directory. Default is the current directory.

-e, --extlib-yml PATH

Path to the .extlib.yml configuration file. Default is <root-path>/.extlib.yml.

-s, --submodule NAME

Name of the external library submodule to process. If not provided, all submodules will be processed.

-v, --version

Show version information.

-h, --help

Show help message.

Commands

mode

Set the operation mode for external libraries.

python arduino-extlib.py mode [OPTIONS] {symlink|copy}

Arguments:

  • symlink: Create symbolic links to submodule files (development mode)

  • copy: Copy files from submodules (distribution mode)

Examples:

# Set all submodules to symlink mode
python arduino-extlib.py mode symlink

# Set specific submodule to copy mode
python arduino-extlib.py -s extras/xensiv-pas-gas-sensor mode copy

# Use custom paths
python arduino-extlib.py -r /path/to/project -e custom-extlib.yml mode symlink

verify

Verify that copied files match their source submodule versions.

python arduino-extlib.py verify [OPTIONS]

This command:

  • Checks if the recorded git tag and hash match the current submodule state

  • Compares copied files with their source files

  • Reports any discrepancies or modifications

  • Exits with error code 1 if verification fails

Examples:

# Verify all submodules
python arduino-extlib.py verify

# Verify specific submodule
python arduino-extlib.py -s extras/xensiv-pas-gas-sensor verify

Configuration File Format

The .extlib.yml file defines how external libraries are integrated. It uses YAML format and supports both single library entries and lists of multiple libraries.

Schema Specification

# Single library configuration
- repo: string              # Required: Path to submodule (relative to root)
  tag: string               # Git tag (auto-updated in copy mode)
  head: string              # Git commit hash (auto-updated in copy mode)
  mode: string              # Current mode: "symlink" or "copy"
  files:                    # File mapping configuration
    dest: string            # Required: Destination path (relative to root)
    src: string|list        # Required: Source files/directories

# Multiple libraries
- repo: extras/lib1
  # ... configuration for lib1 ...
- repo: extras/lib2
  # ... configuration for lib2 ...

Field Descriptions

repo (required)

Path to the external library submodule, relative to the project root directory.

Example: extras/xensiv-pas-gas-sensor

tag (auto-managed)

Git tag of the external library submodule. This field is automatically updated when the mode is set to copy. Set to "unset" in symlink mode.

Example: v1.2.3 or v1.0.0-dirty (if uncommitted changes exist)

head (auto-managed)

Git commit hash of the external library submodule. Automatically updated in copy mode with the full commit hash, potentially suffixed with -dirty if there are uncommitted changes.

Example: fa3b2c1d4e5f67890123456789abcdef12345678 or fa3b2c1d4e5f67890123456789abcdef12345678-dirty

mode (auto-managed)

Current operation mode. Automatically set by the tool:

  • symlink: Files are symbolically linked

  • copy: Files are copied

  • pre-symlink / pre-copy: Intermediate states during mode transitions

files (required)

Defines the file mapping from submodule to destination. Can be a single mapping or a list of mappings.

dest (required)

Destination directory path, relative to the project root where files will be placed.

Example: src/core-c

src (required)

Source files or directories within the submodule. Can be:

  • A single string: src

  • A list of strings: [include, src]

  • Wildcard patterns: *.h (expanded using glob)

Example Configurations

Basic Single Library

- repo: extras/xensiv-pas-gas-sensor
  tag: v1.2.3
  head: fa3b2c1d4e5f67890123456789abcdef12345678
  files:
    dest: src/core
    src: src
  mode: copy

Multiple Files from Same Library

- repo: extras/external-lib
  tag: v2.0.1
  head: 1234567890abcdef1234567890abcdef12345678
  files:
    - dest: src/headers
      src: include
    - dest: src/implementation
      src: src
  mode: copy

Multiple Source Files

- repo: extras/multi-component-lib
  tag: v3.1.0
  head: abcdef1234567890abcdef1234567890abcdef12
  files:
    dest: src/external
    src:
      - core
      - utils
      - drivers
  mode: copy

Development Mode (Symlink)

- repo: extras/active-development-lib
  tag: unset
  head: unset
  files:
    dest: src/dev-lib
    src: src
  mode: symlink

Multiple Libraries

- repo: extras/sensor-lib
  tag: v1.5.2
  head: 11111111222222223333333344444444aaaaaaaa
  files:
    dest: src/sensors
    src: src
  mode: copy

- repo: extras/communication-lib
  tag: v2.1.0
  head: bbbbbbbb5555555566666666777777778888888
  files:
    dest: src/comms
    src: [include, src]
  mode: copy

- repo: extras/utils-lib
  tag: unset
  head: unset
  files:
    dest: src/utils
    src: utilities
  mode: symlink

Workflow Examples

Development Workflow

  1. Initial Setup: Create your .extlib.yml configuration

  2. Development Mode: Set libraries to symlink mode for active development

  3. Testing: Work with live updates from submodules

  4. Pre-release: Switch to copy mode and verify integrity

  5. Release: Distribute with stable copied files

# Start development with symlinks
python arduino-extlib.py mode symlink

# Later, prepare for release
python arduino-extlib.py mode copy

# After committing all changes.
# Verify everything matches
python arduino-extlib.py verify

Version Management

# Check current status
cat .extlib.yml

# Update submodule and switch to copy mode
cd extras/my-submodule
git pull origin main
cd ../..
python arduino-extlib.py mode copy

# This automatically updates tag and head in .extlib.yml

Error Handling

The tool provides comprehensive error handling and logging:

Common Error Scenarios:

  • Missing submodule: Submodule path doesn’t exist

  • Git errors: Unable to read git information from submodule

  • File conflicts: Destination files already exist and are not links/copies

  • Permission errors: Insufficient permissions to create symlinks or copy files

  • Verification failures: Copied files don’t match source or have been modified

Exit Codes:

  • 0: Success

  • 1: Error occurred (verification failure, missing files, etc.)

Troubleshooting

Permission Issues on Windows

Creating symbolic links on Windows may require administrator privileges. Run the command prompt as administrator or use copy mode instead.

Submodule Not Found

Ensure the submodule path in .extlib.yml is correct and the submodule has been initialized:

git submodule update --init --recursive
Verification Failures

If verification fails, check:

  • Whether files have been manually modified in the destination

  • If the submodule has uncommitted changes

  • Whether the recorded tag/hash matches the actual submodule state

File Conflicts

Before switching modes, the tool removes existing destination paths. Ensure you don’t have important files in destination directories that aren’t managed by this tool.