Skip to content

Conversation

@vtomasr5
Copy link
Collaborator

@vtomasr5 vtomasr5 commented May 23, 2025

Changes:

### Performance Improvements

- **[FEATURE]** Add concurrent processing for nested dependencies installation
- **[ENHANCEMENT]** Add retry mechanism for downloading GitHub archives before falling back to git clone
- **[BUGFIX]** Fix concurrency issue when JB_PARALLEL_DOWNLOADS=true was set
- **[ENHANCEMENT]** Refactor the Install function for better maintainability

### Advanced Caching System

- **[FEATURE]** Add global caching system in `~/.cache/jb` to improve performance across projects
- **[FEATURE]** Add cache metadata and indexing for improved organization and retrieval
- **[FEATURE]** Add AWS S3 support for remote caches with full AWS SDK integration
- **[FEATURE]** Implement concurrent cache checking using goroutines for significant performance improvements
- **[FEATURE]** Implement hierarchical cache checking (global → remote) with parallel lookups
- **[FEATURE]** Add cache statistics tracking for performance monitoring
- **[BREAKING]** Remove local cache system in favor of global and remote caches
- **[ENHANCEMENT]** Add new cache management commands with a modern subcommand structure:
  - `jb cache status` - Display cache statistics and health
  - `jb cache flush` - Empty the cache
  - `jb cache add-remote` - Add a remote cache server
  - `jb cache list-remote` - List remote cache servers
  - `jb cache remove-remote` - Remove a remote cache server
  - `jb cache list` - List all cache entries with metadata
- **[ENHANCEMENT]** Add `--no-global-cache` flag to disable global caching
- **[ENHANCEMENT]** Add `JB_CACHE_DIR` environment variable to customize global cache location

### Build and CI Improvements

- **[ENHANCEMENT]** Add arm64 support for macOS binary compilation
- **[ENHANCEMENT]** Enable GitHub Actions for CI/CD
- **[ENHANCEMENT]** Update to support only the latest Go releases
- **[ENHANCEMENT]** Ensure stable transitive dependencies
- **[ENHANCEMENT]** Disable Windows build and ARM 32-bit for Darwin

jb update cache execution times (using Localstack):

export JB_PARALLEL_DOWNLOADS=false # the default

Empty cache:
jb update  16.23s user 5.85s system 11% cpu 3:13.62 total

S3 cached:
jb update  9.98s user 3.18s system 19% cpu 1:08.22 total

All cached:
jb update  10.10s user 3.03s system 18% cpu 1:10.90 total

-------------------------------------

export JB_PARALLEL_DOWNLOADS=true

Empty cache:
jb update  16.35s user 6.15s system 18% cpu 2:01.88 total

S3 cached only:
jb update  11.07s user 3.64s system 115% cpu 12.718 total

All cached:
jb update  10.70s user 3.22s system 99% cpu 14.009 total

To try it out:

git clone https://github.com/closeio/jsonnet-bundler.git -b caching

cd jsonnet-bundler/

make static && cp _output/jb ~/bin/jb

jb --version
v0.6.0-29-gbfa8705

@vtomasr5 vtomasr5 requested a review from a team May 23, 2025 09:09
@vtomasr5 vtomasr5 self-assigned this May 23, 2025
@vtomasr5 vtomasr5 marked this pull request as ready for review May 26, 2025 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants