jsoncs3 share manager does not scale under load

[butonic]

When running a cdperf test agains a deployment in kubernetes I see no errors when running a single api pod:

  █ TOTAL RESULTS 

    checks_total.......................: 9748    25.652572/s
    checks_succeeded...................: 100.00% 9748 out of 9748
    checks_failed......................: 0.00%   0 out of 9748

    ✓ authn -> loginPageResponse - status
    ✓ authn -> authorizationResponse - status
    ✓ authn -> accessTokenResponse - status
    ✓ client -> role.getMyDrives - status
    ✓ client -> search.searchForSharees - status
    ✓ client -> resource.getResourceProperties - status
    ✓ client -> application.createDrive - status
    ✓ client -> resource.createResource - status
    ✓ client -> drive.deactivateDrive - status
    ✓ client -> resource.downloadResource - status
    ✓ client -> tag.getTags - status -- (SKIPPED)
    ✓ client -> tag.createTag - status -- (SKIPPED)
    ✓ client -> drive.deleteDrive - status
    ✓ client -> share.createShare - status
    ✓ client -> share.deleteShare - status
    ✓ client -> resource.deleteResource - status
    ✓ client -> resource.uploadResource - status
    ✓ client -> resource.moveResource - status

    HTTP
    http_req_duration.......................................................: avg=120.86ms min=6.42ms med=40.03ms max=47.6s p(90)=100.52ms p(95)=179.88ms
      { expected_response:true }............................................: avg=120.86ms min=6.42ms med=40.03ms max=47.6s p(90)=100.52ms p(95)=179.88ms
    http_req_failed.........................................................: 0.00%  0 out of 6340
    http_reqs...............................................................: 6340   16.684172/s

    EXECUTION
    iteration_duration......................................................: avg=4.3s     min=1.02s  med=1.04s   max=1m21s p(90)=15.04s   p(95)=15.29s  
    iterations..............................................................: 5324   14.010494/s
    vus.....................................................................: 1      min=0         max=75
    vus_max.................................................................: 75     min=70        max=75

    NETWORK
    data_received...........................................................: 2.0 GB 5.3 MB/s
    data_sent...............................................................: 834 MB 2.2 MB/s




running (6m20.0s), 00/75 VUs, 5324 complete and 4 interrupted iterations
add_remove_tag_100             ✓ [======================================] 0/6 VUs    6m0s
create_remove_group_share_090  ✓ [======================================] 0/6 VUs    6m0s
create_space_080               ✓ [======================================] 0/6 VUs    6m0s
create_upload_rename_delete... ✓ [======================================] 0/6 VUs    6m0s
download_050                   ✓ [======================================] 0/6 VUs    6m0s
navigate_file_tree_020         ✓ [======================================] 0/9 VUs    6m0s
sync_client_110                ✓ [======================================] 00/18 VUs  6m0s
user_group_search_070          ✓ [======================================] 00/18 VUs  6m0s

This is using an nfs noac based rwx storage.

However, when running three pods (yes, and raising the number of VUs) the sharing requests cause the optimistic locking implementation in jsoncs3 to literally give up:

opencloud-api-6d9cfbf48-8q7m9 api {
  "level": "error",
  "service": "sharing",
  "host.name": "opencloud-api-6d9cfbf48-8q7m9",
  "pkg": "rgrpc",
  "traceid": "1178e7d2e777a5504dfe64575087be43",
  "hostname": "opencloud-api-6d9cfbf48-8q7m9",
  "userID": "6ce35cdc-b0b7-1040-8871-1ff3cb45568a",
  "shareID": "10cb695a-ecee-4cdc-a817-d9f5c5630ea8:67868a20-b0b7-1040-850b-1ff3cb45568a:00f1e3c8-e597-4f1d-98e1-1bc8b5936fc4",
  "error": "internal error: /data/spaces/jsoncs3-share-manager-metadata!jsoncs3-share-manager-metadata/groups/6ce35cdc-b0b7-1040-8871-1ff3cb45568a/received.json",
  "time": "2026-03-10T11:33:11Z",
  "message": "persisting added share failed. giving up.",
}

The shares are all done to the same group, causing congestion on the received.json file. This is not likely to happen in production environments, but we need to find a better solution here, as it leads to test failures:

  █ TOTAL RESULTS 

    checks_total.......................: 31648  81.586693/s
    checks_succeeded...................: 99.67% 31544 out of 31648
    checks_failed......................: 0.32%  104 out of 31648

    ✓ authn -> loginPageResponse - status
    ✓ authn -> authorizationResponse - status
    ✓ authn -> accessTokenResponse - status
    ✓ client -> search.searchForSharees - status
    ✓ client -> role.getMyDrives - status
    ✓ client -> resource.getResourceProperties - status
    ✓ client -> application.createDrive - status
    ✓ client -> resource.createResource - status
    ✓ client -> drive.deactivateDrive - status
    ✓ client -> resource.downloadResource - status
    ✓ client -> tag.getTags - status -- (SKIPPED)
    ✓ client -> tag.createTag - status -- (SKIPPED)
    ✓ client -> drive.deleteDrive - status
    ✗ client -> share.createShare - status
      ↳  78% — ✓ 130 / ✗ 36
    ✗ client -> share.deleteShare - status
      ↳  59% — ✓ 98 / ✗ 68
    ✓ client -> resource.deleteResource - status
    ✓ client -> resource.uploadResource - status
    ✓ client -> resource.moveResource - status

    HTTP
    http_req_duration.......................................................: avg=557.21ms min=4.28ms med=318.81ms max=1m6s  p(90)=707.97ms p(95)=1.38s 
      { expected_response:true }............................................: avg=544.86ms min=4.28ms med=318.37ms max=1m6s  p(90)=688.24ms p(95)=1.32s 
    http_req_failed.........................................................: 0.49%  104 out of 20806
    http_reqs...............................................................: 20806  53.636651/s

    EXECUTION
    iteration_duration......................................................: avg=5.36s    min=1.05s  med=1.39s    max=1m44s p(90)=15.37s   p(95)=16.84s
    iterations..............................................................: 17167  44.255522/s
    vus.....................................................................: 1      min=0            max=300
    vus_max.................................................................: 300    min=80           max=300

    NETWORK
    data_received...........................................................: 7.9 GB 20 MB/s
    data_sent...............................................................: 2.7 GB 6.9 MB/s




running (6m27.9s), 000/300 VUs, 17167 complete and 6 interrupted iterations
add_remove_tag_100             ✓ [======================================] 00/24 VUs  6m0s
create_remove_group_share_090  ✓ [======================================] 00/24 VUs  6m0s
create_space_080               ✓ [======================================] 00/24 VUs  6m0s
create_upload_rename_delete... ✓ [======================================] 00/24 VUs  6m0s
download_050                   ✓ [======================================] 00/24 VUs  6m0s
navigate_file_tree_020         ✓ [======================================] 00/36 VUs  6m0s
sync_client_110                ✓ [======================================] 00/72 VUs  6m0s
user_group_search_070          ✓ [======================================] 00/72 VUs  6m0s

Since we were discussing touching the code and using, e.g. pessimistic locking I doubt the problem will go away. Our tests are hitting a corner case ... and hard.

I think we should consider a nats jetstream based solution that uses workers to batch writes to the same files. @aduffeck @rhafer please see summary of a discussion with an AI below. I'm not as pumped about the approach as qwen3.5-27b but still.

AI discussion

Feel free to ignore AI content, but It does explain the approach I have in mind quite well.

You're describing a synchronous API over asynchronous infrastructure pattern — essentially implementing strong consistency guarantees while leveraging the throughput benefits of eventual consistency internally. This is an excellent approach that gives you the best of both worlds.

Here's how to implement it using NATS JetStream with a back-channel completion mechanism:

---

Architecture Overview

┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  jsoncs3    │────▶│ NATS JetStream   │◀────│  jsoncs3         │
│  Instance A │     │ (Write Queue)    │     │  Instance B      │
└─────────────┘     └──────────────────┘     └──────────────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │ Write Workers       │
              │ (Process & Persist) │
              └─────────────────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │ Completion Channel  │
              │ (Back channel for   │
              │  caller to wait)    │
              └─────────────────────┘

---

Key Components

1. Request ID & Completion Tracking

Each write operation gets a unique request ID that's tracked through the system:

// pkg/share/manager/jsoncs3/writequeue/types.go

type WriteOperation struct {
    RequestID string `json:"request_id"`      // Unique per API call
    UserID    string `json:"userid"`          // User/group ID for routing
    ShareID   string `json:"shareid"`         // Share being added/removed
    Operation int    `json:"operation"`       // 0=Add, 1=Remove
    Timestamp time.Time `json:"timestamp"`
}

type CompletionEvent struct {
    RequestID string `json:"request_id"`
    Success   bool   `json:"success"`
    Error     string `json:"error,omitempty"`
}

2. Modified sharecache with Async Write + Sync Wait

// sharecache.go modifications

type Cache struct {
    lockMap sync.Map
    UserShares mtimesyncedcache.Map[string, *UserShareCache]
    storage   metadata.Storage
    namespace string
    filename  string
    ttl       time.Duration
    
    // NATS JetStream for async writes and completion signaling
    js jetstream.JetStreamContext
    queueGroup string
}

func New(s metadata.Storage, namespace, filename string, ttl time.Duration) Cache {
    return Cache{
        UserShares: mtimesyncedcache.Map[string, *UserShareCache]{},
        storage:    s,
        namespace:  namespace,
        filename:   filename,
        ttl:        ttl,
        lockMap:    sync.Map{},
        // js and queueGroup initialized via config or dependency injection
    }
}

func (c *Cache) Add(ctx context.Context, userid, shareID string) error {
    ctx, span := appctx.GetTracerProvider(ctx).Tracer(tracerName).Start(ctx, "Add")
    defer span.End()

    // Generate unique request ID for this operation
    requestID := uuid.New().String()

    // 1. Update in-memory cache immediately (for read consistency)
    c.initializeIfNeeded(userid, "")
    us, _ := c.UserShares.LoadOrStore(userid, &UserShareCache{
        UserShares: map[string]*SpaceShareIDs{},
    })
    
    storageid, spaceid, _ := shareid.Decode(shareID)
    ssid := storageid + shareid.IDDelimiter + spaceid
    
    if us.UserShares[ssid] == nil {
        us.UserShares[ssid] = &SpaceShareIDs{IDs: map[string]struct{}{}}
    }
    us.UserShares[ssid].IDs[shareID] = struct{}{}

    // 2. Queue the write operation for durable persistence
    if c.js != nil {
        op := WriteOperation{
            RequestID: requestID,
            UserID:    userid,
            ShareID:   shareID,
            Operation: OpAddShare,
            Timestamp: time.Now(),
        }
        
        data, _ := json.Marshal(op)
        _, err := c.js.Publish(ctx, "sharecache.write."+userid, data)
        if err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "failed to queue write operation")
        }

        // 3. Wait for completion via back channel (synchronous API guarantee)
        if err := c.waitForCompletion(ctx, requestID); err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "write operation failed")
        }
    } else {
        // Fallback to synchronous persist if NATS not configured
        if err := c.Persist(ctx, userid); err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "failed to persist share")
        }
    }

    span.SetStatus(codes.Ok, "")
    return nil
}

func (c *Cache) Remove(ctx context.Context, userid, shareID string) error {
    ctx, span := appctx.GetTracerProvider(ctx).Tracer(tracerName).Start(ctx, "Remove")
    defer span.End()

    requestID := uuid.New().String()

    // 1. Update in-memory cache immediately
    us, loaded := c.UserShares.LoadOrStore(userid, &UserShareCache{
        UserShares: map[string]*SpaceShareIDs{},
    })
    
    if loaded {
        storageid, spaceid, _ := shareid.Decode(shareID)
        ssid := storageid + shareid.IDDelimiter + spaceid
        
        if us.UserShares[ssid] != nil {
            delete(us.UserShares[ssid].IDs, shareID)
        }
    }

    // 2. Queue the write operation for durable persistence
    if c.js != nil {
        op := WriteOperation{
            RequestID: requestID,
            UserID:    userid,
            ShareID:   shareID,
            Operation: OpRemoveShare,
            Timestamp: time.Now(),
        }
        
        data, _ := json.Marshal(op)
        _, err := c.js.Publish(ctx, "sharecache.write."+userid, data)
        if err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "failed to queue write operation")
        }

        // 3. Wait for completion via back channel (synchronous API guarantee)
        if err := c.waitForCompletion(ctx, requestID); err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "write operation failed")
        }
    } else {
        // Fallback to synchronous persist if NATS not configured
        if err := c.Persist(ctx, userid); err != nil {
            span.RecordError(err)
            return errors.Wrap(err, "failed to persist removal")
        }
    }

    span.SetStatus(codes.Ok, "")
    return nil
}

// waitForCompletion blocks until the operation completes or times out
func (c *Cache) waitForCompletion(ctx context.Context, requestID string) error {
    // Create a subscription for this specific request ID on the completion channel
    sub, err := c.js.SubscribeSync("sharecache.complete."+requestID)
    if err != nil {
        return errors.Wrap(err, "failed to subscribe to completion channel")
    }
    defer sub.Unsubscribe()

    // Set up a timeout for waiting (e.g., 5 seconds max)
    ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    // Wait for the completion message
    msg, err := sub.NextMsgWithContext(ctx)
    if err != nil {
        return errors.Wrap(err, "timeout waiting for operation completion")
    }

    var event CompletionEvent
    if err := json.Unmarshal(msg.Data(), &event); err != nil {
        return errors.Wrap(err, "failed to unmarshal completion event")
    }

    // Verify this is the right request ID (should always be)
    if event.RequestID != requestID {
        return fmt.Errorf("unexpected request ID in completion: %s", event.RequestID)
    }

    // Return error if operation failed
    if !event.Success {
        return errors.New(event.Error)
    }

    return nil
}

3. Write Worker with Completion Signaling

// pkg/share/manager/jsoncs3/writequeue/worker.go

type WriteWorker struct {
    js jetstream.JetStreamContext
    cache *sharecache.Cache
    queueGroup string
}

func NewWriteWorker(js jetstream.JetStreamContext, cache *sharecache.Cache) *WriteWorker {
    return &WriteWorker{
        js:       js,
        cache:    cache,
        queueGroup: "sharecache-writer", // Queue group for load balancing
    }
}

func (w *WriteWorker) Start(ctx context.Context) error {
    log := appctx.GetLogger(ctx).With().Str("component", "writequeue-worker").Logger()
    
    // Subscribe to all write operations with a queue group
    sub, err := w.js.PullSubscribe("sharecache.write.*", w.queueGroup)
    if err != nil {
        return errors.Wrap(err, "failed to subscribe to write queue")
    }

    log.Info().Msg("write worker started")

    // Process messages in batches for efficiency
    go func() {
        ticker := time.NewTicker(100 * time.Millisecond)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                w.processBatch(ctx, sub, log)
            }
        }
    }()

    return nil
}

func (w *WriteWorker) processBatch(ctx context.Context, sub jetstream.PullSubscription, log logger.Logger) {
    // Fetch up to 100 messages at a time for efficiency
    msgs, err := sub.Fetch(100, jetstream.MaxWait(2*time.Second))
    if len(msgs) == 0 || (err != nil && !errors.Is(err, context.DeadlineExceeded)) {
        return
    }

    // Group messages by user ID for serialization
    userOps := make(map[string][]*jetstream.Msg)
    for _, msg := range msgs {
        var op WriteOperation
        if err := json.Unmarshal(msg.Data(), &op); err != nil {
            log.Error().Err(err).Msg("failed to unmarshal write operation")
            continue
        }
        userOps[op.UserID] = append(userOps[op.UserID], msg)
    }

    // Process each user's operations sequentially (no concurrent writes per user)
    for userID, ops := range userOps {
        w.processUserOperations(ctx, userID, ops, log)
    }
}

func (w *WriteWorker) processUserOperations(ctx context.Context, userID string, msgs []*jetstream.Msg, log logger.Logger) {
    // Acquire local lock for this user to ensure serialization within this instance
    unlock := w.cache.lockUser(userID)
    defer unlock()

    // Sync with server state first (optimistic locking base)
    if err := w.cache.syncWithLock(ctx, userID); err != nil {
        log.Error().Err(err).Str("userid", userID).Msg("failed to sync before processing")
        return
    }

    // Apply all operations for this user sequentially
    for _, msg := range msgs {
        var op WriteOperation
        if err := json.Unmarshal(msg.Data(), &op); err != nil {
            log.Error().Err(err).Str("userid", userID).Msg("failed to unmarshal operation")
            w.publishCompletion(op.RequestID, false, "unmarshal error")
            continue
        }

        // Apply the operation to in-memory cache (already done by caller, but ensure consistency)
        switch op.Operation {
        case OpAddShare:
            w.cache.initializeIfNeeded(userID, "")
            us, _ := w.cache.UserShares.LoadOrStore(userID, &sharecache.UserShareCache{
                UserShares: map[string]*sharecache.SpaceShareIDs{},
            })
            
            storageid, spaceid, _ := shareid.Decode(op.ShareID)
            ssid := storageid + shareid.IDDelimiter + spaceid
            
            if us.UserShares[ssid] == nil {
                us.UserShares[ssid] = &sharecache.SpaceShareIDs{IDs: map[string]struct{}{}}
            }
            us.UserShares[ssid].IDs[op.ShareID] = struct{}{}

        case OpRemoveShare:
            if us, loaded := w.cache.UserShares.Load(userID); loaded {
                storageid, spaceid, _ := shareid.Decode(op.ShareID)
                ssid := storageid + shareid.IDDelimiter + spaceid
                
                if us.UserShares[ssid] != nil {
                    delete(us.UserShares[ssid].IDs, op.ShareID)
                }
            }
        }

        // Persist to storage with optimistic locking (retry on conflict)
        persistErr := w.cache.Persist(ctx, userID)
        
        if persistErr != nil {
            log.Error().Err(persistErr).Str("userid", userID).Msg("failed to persist")
            w.publishCompletion(op.RequestID, false, persistErr.Error())
            msg.Nak() // Re-queue for retry
            continue
        }

        // Publish completion event on back channel
        w.publishCompletion(op.RequestID, true, "")
        
        // Acknowledge successful processing
        msg.Ack()
    }
}

// publishCompletion sends a completion event to the back channel
func (w *WriteWorker) publishCompletion(requestID string, success bool, errorMsg string) {
    event := CompletionEvent{
        RequestID: requestID,
        Success:   success,
        Error:     errorMsg,
    }
    
    data, _ := json.Marshal(event)
    _, err := w.js.Publish(context.Background(), "sharecache.complete."+requestID, data)
    if err != nil {
        appctx.GetLogger(context.Background()).Error().Err(err).Msg("failed to publish completion event")
    }
}

4. NATS JetStream Stream Configuration

# nats-server config or via API

streams:
  - name: "sharecache-operations"
    subjects: ["sharecache.write.*"]
    storage: file                 # File-backed for durability
    max_bytes: 1073741824         # 1GB retention
    discard_policy: old           # Drop oldest if full
    
  - name: "sharecache-completions"
    subjects: ["sharecache.complete.*"]
    storage: memory               # Memory-backed (short-lived)
    max_msgs_per_subject: 1       # Only one completion per request ID
    max_age: 60s                  # Auto-expire after 60 seconds

queue_groups:
  - name: "sharecache-writer"
    stream: "sharecache-operations"

---

Benefits of This Approach

Feature	Optimistic Locking	Pessimistic Locking	Async + Sync Wait
Consistency	Eventual (retry window)	Strong (synchronous)	Strong (synchronous API)
Data Loss Risk	Low	Zero	Zero (JetStream durable)
Latency	Medium (retries add delay)	High (lock acquisition)	Low (async + wait)
Throughput	Medium (contention causes retries)	Low (serialized writes)	High (batched async processing)
Scalability	Poor (thundering herd)	Poor (single lock per user)	Excellent (queue groups distribute load)

---

Key Advantages

1. Strong Consistency: API returns only after operation is persisted and visible to subsequent Get calls
2. No Data Loss: NATS JetStream stores messages durably until acknowledged
3. Better Throughput: Writes are batched and processed asynchronously, reducing I/O contention
4. Reduced Lock Contention: Only one worker processes writes per user ID at a time via Queue Groups
5. Automatic Crash Recovery: If a node crashes, queued messages remain in JetStream and are processed by another instance
6. Timeout Protection: Callers don't wait forever; 5-second timeout prevents hanging

---

Configuration Recommendations for Production

# NATS JetStream Stream Configuration
stream:
  name: "sharecache-operations"
  subjects: ["sharecache.write.*"]
  storage: file
  max_bytes: 1073741824 # 1GB retention
  discard_policy: old

completion_stream:
  name: "sharecache-completions"
  subjects: ["sharecache.complete.*"]
  storage: memory
  max_msgs_per_subject: 1
  max_age: 60s

# Queue Group for Load Balancing
queue_group: "sharecache-writer"

# Worker Configuration
max_batch_size: 100
batch_timeout: 100ms
retry_attempts: -1 # Infinite retry (JetStream handles this)

# Wait Timeout Configuration
wait_timeout: 5s   # Max time to wait for completion

---

Summary

This async infrastructure with synchronous API approach provides:

• ✅ Strong consistency — no eventual consistency window visible to callers
• ✅ Zero data loss on crash — JetStream durability + message acknowledgment
• ✅ Better throughput — batched async processing reduces I/O contention
• ✅ Reduced lock contention — queue groups distribute load across instances
• ✅ Automatic recovery — queued messages persist and are processed by other nodes
• ✅ Timeout protection — callers don't wait forever

This is the recommended solution for your use case where atomicity and data integrity are critical, but you need to handle high write contention across multiple instances while maintaining a synchronous API contract.

[aduffeck]

I understand the idea behind the nats-based batching approach, but I really don't like its complexity for a "simple" task like persisting the shares.

Personally I would prefer to evaluate a much simpler approach of using a local (rwx-)storage with write-locks when writing the data. We could then put a NATS cache on top of it to make reads faster. That would match what we do with the file metadata and should help with the congestion issues we currently see with the current CS3 storage approach.

[butonic]

@aduffeck another approach would be to use directories with one json file per share instead of a list of shares inside a single json file. that would get rid of most of the locks. However the server has to build an in memory cache to efficiently list shares, which would require invalidating the cache based on the etag of the directory ... but that requires etag propagation / tree time propagation, which is not enabled on the system storage ... and it would have to be synchronous ...

Regarding the json implementation with local disk storage plus busy locking. That does not reduce the number of locks we have to make.

We could use nats as the cache and also use it for all reads. writing goes to disk as well as nats. and on disk we always write individual files. no locking of a json file necessary when adding or removing shares. when a jsoncs3 share manager starts it iterates over all files and checks if they are in nats (they have a change time so we can even identify if the nats entry is more recent. We only need to update the nats entry if the on disk version is more recent. how do we identify if an entry has been deleted on disk but not in nats? well, it would show up in listings and users can just re delete it?). we could even watch the nats kv and use workers to write individual files on disk.

Hm ... we could even use local (rwo) storage with this and use a nats watcher to persist the data on all nodes. When a jsoncs3 manager starts it will fetch the whole nats kv bucket and write individual files to disk ... well. why rely on the cs3 api then if we synchronize the data using nats. we can use plain json with local storage and a single local writer ... that can even batches events ... and suddenly there is no locking and batched writes ...

can we really switch to individual files?

let me check the storage json...

❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- ls storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/js/on/cs/3-/share-manager-metadata/storages
a3346854-f557-4a90-8ab8-758903c7410b
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- ls -l storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/js/on/cs/3-/share-manager-metadata/storages/a3346854-f557-4a90-8ab8-758903c7410b/
total 0
lrwxrwxrwx    1 opencloud-user nobody          55 Mar 24 13:03 050c0164-b25f-1040-9ae1-6772d100cb96.json -> ../../../../../11/21/00/ec/-2b18-4b56-8a1f-81065aec6f5b
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- strings storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/11/21/00/ec/-2b18-4b56-8a1f-81065aec6f5b.mpk
user.oc.name
)050c0164-b25f-1040-9ae1-6772d100cb96.json
user.oc.blobsize
4356
user.oc.id
$112100ec-2b18-4b56-8a1f-81065aec6f5b
user.oc.type
user.oc.parentid
$328cde19-5dca-406c-8986-b2b7ccf33bd6
user.oc.blobid
$4c6a39f6-b384-43c5-a733-a919ba36523f
user.oc.cs.md5
user.oc.cs.sha1
user.oc.cs.adler32
user.oc.mtime
2026-03-24T13:04:40.740141566Z
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- cat storage/metadata/spaces/js/oncs3-share-manager-metadata/blobs/4c/6a/39/f6/-b384-43c5-a733-a919ba36523f | jq
{
  "Shares": {
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:1b02afc0-1d38-4295-a42c-707c7ea5faf9": {
      "id": {
        "opaque_id": "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:1b02afc0-1d38-4295-a42c-707c7ea5faf9"
      },
      "resource_id": {
        "storage_id": "a3346854-f557-4a90-8ab8-758903c7410b",
        "opaque_id": "818cb717-bbeb-4f79-a7c7-cb223b3130c1",
        "space_id": "050c0164-b25f-1040-9ae1-6772d100cb96"
      },
      "permissions": {
        "permissions": {
          "get_path": true,
          "get_quota": true,
          "initiate_file_download": true,
          "list_container": true,
          "list_recycle": true,
          "stat": true
        }
      },
      "grantee": {
        "type": 1,
        "Id": {
          "UserId": {
            "idp": "https://keycloak.opencloud.test/realms/openCloud",
            "opaque_id": "050f4d2e-b25f-1040-9ae3-6772d100cb96",
            "type": 1
          }
        }
      },
      "owner": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "creator": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "ctime": {
        "seconds": 1774357475,
        "nanos": 535994074
      },
      "mtime": {
        "seconds": 1774357475,
        "nanos": 535994074
      }
    },
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:2eccf063-7282-4e31-9764-e452ad420fad": {
      "id": {
        "opaque_id": "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:2eccf063-7282-4e31-9764-e452ad420fad"
      },
      "resource_id": {
        "storage_id": "a3346854-f557-4a90-8ab8-758903c7410b",
        "opaque_id": "38544b3c-5794-4f30-a1a7-66a0c836709c",
        "space_id": "050c0164-b25f-1040-9ae1-6772d100cb96"
      },
      "permissions": {
        "permissions": {
          "get_path": true,
          "get_quota": true,
          "initiate_file_download": true,
          "list_container": true,
          "list_recycle": true,
          "stat": true
        }
      },
      "grantee": {
        "type": 1,
        "Id": {
          "UserId": {
            "idp": "https://keycloak.opencloud.test/realms/openCloud",
            "opaque_id": "050f4d2e-b25f-1040-9ae3-6772d100cb96",
            "type": 1
          }
        }
      },
      "owner": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "creator": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "ctime": {
        "seconds": 1774357447,
        "nanos": 545535360
      },
      "mtime": {
        "seconds": 1774357447,
        "nanos": 545535360
      }
    },
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:8a44a6f3-776d-42b8-8601-a7e402768722": {
      "id": {
        "opaque_id": "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:8a44a6f3-776d-42b8-8601-a7e402768722"
      },
      "resource_id": {
        "storage_id": "a3346854-f557-4a90-8ab8-758903c7410b",
        "opaque_id": "28bce249-47ab-45b1-9830-008fc5d19d6b",
        "space_id": "050c0164-b25f-1040-9ae1-6772d100cb96"
      },
      "permissions": {
        "permissions": {
          "get_path": true,
          "get_quota": true,
          "initiate_file_download": true,
          "list_container": true,
          "list_recycle": true,
          "stat": true
        }
      },
      "grantee": {
        "type": 1,
        "Id": {
          "UserId": {
            "idp": "https://keycloak.opencloud.test/realms/openCloud",
            "opaque_id": "050f4d2e-b25f-1040-9ae3-6772d100cb96",
            "type": 1
          }
        }
      },
      "owner": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "creator": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "ctime": {
        "seconds": 1774357480,
        "nanos": 659501347
      },
      "mtime": {
        "seconds": 1774357480,
        "nanos": 659501347
      }
    },
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:ec65a091-697d-4d3a-b755-fa1f71a1054e": {
      "id": {
        "opaque_id": "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:ec65a091-697d-4d3a-b755-fa1f71a1054e"
      },
      "resource_id": {
        "storage_id": "a3346854-f557-4a90-8ab8-758903c7410b",
        "opaque_id": "97d0b0cb-b185-4b61-81e9-ff3db0a35cf4",
        "space_id": "050c0164-b25f-1040-9ae1-6772d100cb96"
      },
      "permissions": {
        "permissions": {
          "get_path": true,
          "get_quota": true,
          "initiate_file_download": true,
          "list_container": true,
          "list_recycle": true,
          "stat": true
        }
      },
      "grantee": {
        "type": 1,
        "Id": {
          "UserId": {
            "idp": "https://keycloak.opencloud.test/realms/openCloud",
            "opaque_id": "050f4d2e-b25f-1040-9ae3-6772d100cb96",
            "type": 1
          }
        }
      },
      "owner": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "creator": {
        "idp": "https://keycloak.opencloud.test/realms/openCloud",
        "opaque_id": "050c0164-b25f-1040-9ae1-6772d100cb96",
        "type": 1
      },
      "ctime": {
        "seconds": 1774357437,
        "nanos": 49005869
      },
      "mtime": {
        "seconds": 1774357437,
        "nanos": 49005869
      }
    }
  },
  "Etag": "\"ee8111f58ceff9c7608a4d575b7848c8\""
}

-> kv entries of shares in a json object with an etag.

user created.json:

❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- ls -l storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/js/on/cs/3-/share-manager-metadata/users/050c0164-b25f-1040-9ae1-6772d100cb96/
total 0
lrwxrwxrwx    1 opencloud-user nobody          55 Mar 24 13:03 created.json -> ../../../../../44/62/00/11/-595d-4e1c-99fa-cbac0fe33707
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- strings storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/44/62/00/11/-595d-4e1c-99fa-cbac0fe33707.mpk
user.oc.type
user.oc.name
created.json
user.oc.cs.adler32
user.oc.cs.sha1
user.oc.mtime
2026-03-24T13:04:40.869149886Z
user.oc.blobsize
user.oc.cs.md5
user.oc.id
$44620011-595d-4e1c-99fa-cbac0fe33707
user.oc.blobid
$25f35d00-3793-49b5-9da9-3cc037040ac4
user.oc.parentid
$61a0f896-3038-4484-880f-fbc3f86785aa                                                                               
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- cat storage/metadata/spaces/js/oncs3-share-manager-metadata/blobs/25/f3/5d/00/-3793-49b5-9da9-3cc037040ac4 | jq
{
  "UserShares": {
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96": {
      "IDs": {
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:1b02afc0-1d38-4295-a42c-707c7ea5faf9": {},
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:2eccf063-7282-4e31-9764-e452ad420fad": {},
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:8a44a6f3-776d-42b8-8601-a7e402768722": {},
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:ec65a091-697d-4d3a-b755-fa1f71a1054e": {}
      }
    }
  },
  "Etag": "\"2c701e93a17f5723b2b4233b2d28ed9b\""
}

-> a per space map of shareid -> empty object to check presence

received.json:

❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- ls -l storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/js/on/cs/3-/share-manager-metadata/users/050f4d2e-b25f-1040-9ae3-6772d100cb96/
total 0
lrwxrwxrwx    1 opencloud-user nobody          55 Mar 24 13:03 received.json -> ../../../../../65/07/3d/4d/-8734-4eeb-a6ec-79fb1020df60
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- strings storage/metadata/spaces/js/oncs3-share-manager-metadata/nodes/65/07/3d/4d/-8734-4eeb-a6ec-79fb1020df60.mpk
user.oc.type
user.oc.cs.sha1
user.oc.name
received.json
user.oc.blobsize
user.oc.parentid
$554d4821-4f5a-45fb-b85f-b0b1ddcbb7d7
user.oc.cs.adler32
user.oc.cs.md5
user.oc.blobid
$e0a384df-2ec2-442c-bd22-742e71345544
user.oc.id
$65073d4d-8734-4eeb-a6ec-79fb1020df60
user.oc.mtime
2026-03-24T13:04:41.300237653Z
❯ kubectl -n opencloud exec -c api opencloud-api-7884d65646-dsjct -- cat storage/metadata/spaces/js/oncs3-share-manager-metadata/blobs/e0/a3/84/df/-2ec2-442c-bd22-742e71345544 | jq
{
  "Spaces": {
    "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96": {
      "States": {
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:1b02afc0-1d38-4295-a42c-707c7ea5faf9": {
          "State": 2,
          "MountPoint": {
            "path": "YYYY-MM-DD Deployment OpenCloud(1).drawio"
          },
          "Hidden": false
        },
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:2eccf063-7282-4e31-9764-e452ad420fad": {
          "State": 2,
          "MountPoint": {
            "path": "2026-03-23 Deployment OpenCloud.drawio"
          },
          "Hidden": false
        },
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:8a44a6f3-776d-42b8-8601-a7e402768722": {
          "State": 2,
          "MountPoint": {
            "path": "2026-03-23 Deployment OpenCloud.drawio.pdf"
          },
          "Hidden": false
        },
        "a3346854-f557-4a90-8ab8-758903c7410b:050c0164-b25f-1040-9ae1-6772d100cb96:ec65a091-697d-4d3a-b755-fa1f71a1054e": {
          "State": 2,
          "MountPoint": {
            "path": "2026-02-06 Deployment OpenCloud.drawio"
          },
          "Hidden": false
        }
      }
    }
  }
}

-> a per space list of shares and some properties

Yep, the lists of shares can be stored in individual files. The created.json is redundant and could be populated as a dedicated list in nats with {userid}.created.{shareid} entries that can then be queried using wildcards {userid}.created.*. then we can fetch the created entries from nats.

No locking necessary, AFAICT. Hm ... for easier backup we can require an rwx storage, then the data on disk is the same on all pods. however, concurrent requests may want to update the same share so we would need locking there. but that could use a busy wait lock. Or we decide to use RWO storage ... but then the path for the share manager would have to use a different storage class ... possible ... but maybe it is safer to just always use locking ...

[butonic]

hm ... but how do we prevent overwriting a share when it is updated concurrently

👀

ah, nats supports optimistic locks:

kv.Update(ctx, "mykey", value, expectedRevision)

[butonic]

Another Idea would be to use a json backend implementation that uses the local filesystem and the same locking mechanism we use in the storage drivers. That would allow us to wait for a filelock instead of the optimistic locking approach. This would alos work on network filesystems, just like posix.

We could even retire the storage-system service and always directly write to disk and cache things in nats. maybe even use nats to coordinate locking, so we do not stress the filesystem as much. Tracked in #3058

[butonic]

While looking into this I gave all the information ta qwen 3.8 27 and let it chew on this while I was at 🥋🤼

It came up with opencloud-eu/reva#798 but I'll discuss the approach with @aduffeck and @rhafer first.

[butonic]

aaaand another idea just proposed by @rhafer why do we use the event mechanism to autoaccept shares. That could be done by the share manager directly... hm maybe because the request then has to wait for all file to be written? be we can offload that to a goroutine and return early. if there are errors we have the same as with the current event based implementation. ... with nats we do have an ack mechanism though... 🤔