Local Git repositories

Sourcebot can sync code from generic git repositories stored in a local directory. This can be helpful in scenarios where you already have a large number of repos already checked out. Local repositories are treated as read-only, meaning Sourcebot will not git fetch new revisions.

If you’re not familiar with Sourcebot connections, please read that overview first.

Getting Started

Only folders containing git repositories at their root and have a remote.origin.url set in their git config are supported at this time. All other folders will be skipped.

Let’s assume we have a repos directory located at $(PWD) with a collection of git repositories:

repos/
├─ repo_1/
├─ repo_2/
├─ repo_3/
├─ ...

To get Sourcebot to index these repositories:

1. Mount a volume

   We need to mount a docker volume to the repos directory so Sourcebot can read it’s contents. Sourcebot will not write to local repositories, so we can mount a separate read-only volume:

   docker run \
       -v $(pwd)/repos:/repos:ro \
       /* additional args */ \
       docker.sourcebot.dev/sourcebot-dev/sourcebot:latest
   
2. Create a connection

   We can now create a new git connection, specifying local paths with the file:// prefix. Glob patterns are supported. For example:

   {
       "type": "git",
       "url": "file:///repos/*"
   }
   

   Sourcebot will expand this glob pattern into paths /repos/repo_1, /repos/repo_2, etc. and index all valid git repositories.

Examples

Sync individual repo

{
    "type": "git",
    "url": "file:///path/to/git_repo"
}

Sync multiple repos using glob patterns

// Attempt to sync directories contained in `repos/` (non-recursive)
{
    "type": "git",
    "url": "file:///repos/*"
}

Optimizing git operations

Sourcebot performs a number of operations that require traversing a repository’s entire commit history (e.g., git rev-list --count HEAD). These operations can be slow in repositories with a large number of commits.

Typically, a commit graph is generated to speed up these operations (see #791). However, since local repositories are treated as read-only, Sourcebot will not generate a commit graph for them.

A commit graph can be manually generated by running the following command in the repository’s root directory:

git commit-graph write --reachable

The commit graph can be updated when fetching with --write-commit-graph:

git fetch --write-commit-graph

Schema reference

Reference

schemas/v3/genericGitHost.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "title": "GenericGitHostConnectionConfig",
  "properties": {
    "type": {
      "const": "git",
      "description": "Generic Git host configuration"
    },
    "url": {
      "type": "string",
      "format": "url",
      "description": "The URL to the git repository. This can either be a remote URL (prefixed with `http://` or `https://`) or a absolute path to a directory on the local machine (prefixed with `file://`). If a local directory is specified, it must point to the root of a git repository. Local directories are treated as read-only modified. Local directories support glob patterns.",
      "pattern": "^(https?:\\/\\/[^\\s/$.?#].[^\\s]*|file:\\/\\/\\/[^\\s]+)$",
      "examples": [
        "https://github.com/sourcebot-dev/sourcebot",
        "file:///path/to/repo",
        "file:///repos/*"
      ]
    },
    "revisions": {
      "type": "object",
      "description": "The revisions (branches, tags) that should be included when indexing. The default branch (HEAD) is always indexed. A maximum of 64 revisions can be indexed, with any additional revisions being ignored.",
      "properties": {
        "branches": {
          "type": "array",
          "description": "List of branches to include when indexing. For a given repo, only the branches that exist on the repo's remote *and* match at least one of the provided `branches` will be indexed. The default branch (HEAD) is always indexed. Glob patterns are supported. A maximum of 64 branches can be indexed, with any additional branches being ignored.",
          "items": {
            "type": "string"
          },
          "examples": [
            [
              "main",
              "release/*"
            ],
            [
              "**"
            ]
          ],
          "default": []
        },
        "tags": {
          "type": "array",
          "description": "List of tags to include when indexing. For a given repo, only the tags that exist on the repo's remote *and* match at least one of the provided `tags` will be indexed. Glob patterns are supported. A maximum of 64 tags can be indexed, with any additional tags being ignored.",
          "items": {
            "type": "string"
          },
          "examples": [
            [
              "latest",
              "v2.*.*"
            ],
            [
              "**"
            ]
          ],
          "default": []
        }
      },
      "additionalProperties": false
    },
    "enforcePermissions": {
      "type": "boolean",
      "description": "Controls whether repository permissions are enforced for this connection. When `PERMISSION_SYNC_ENABLED` is false, this setting has no effect. Defaults to the value of `PERMISSION_SYNC_ENABLED`. See https://docs.sourcebot.dev/docs/features/permission-syncing"
    },
    "enforcePermissionsForPublicRepos": {
      "type": "boolean",
      "default": false,
      "description": "Controls whether repository permissions are enforced for public repositories in this connection. When true, public repositories are only visible to users with a linked account for this connection's code host. When false, public repositories are visible to all users. Has no effect when enforcePermissions is false. Defaults to false. See https://docs.sourcebot.dev/docs/features/permission-syncing"
    }
  },
  "required": [
    "type",
    "url"
  ],
  "additionalProperties": false
}