Skip to content

GitLab

"product/ERP5/interfaces/metadata_discoverable.py" did not exist on "master"
Closed
Created by Cédric Le Ninivin@cedric.leninivinOwner8 of 14 tasks completed8/14 tasks
Report abuse

Draft: Cdn requester

Notes from initial discussion:

So domain name validation should be done in a separate SR that request through a remote node to the Node.

SR CDN is purely technical now called "technical CDN". New SR is business is doing a lot a validation.

Premium CDN existing Shared instance are all ported to new SR.

Remove everyone from premium CDN project aside from its operators.

New SR can use Jinja with Buildout.

At the moment the Business CDN SR will only do the Domain name validation.

XXX Maybe we want to sign the parameters provided by the Business CDN to make sure we trust the source. After cleaning up the current data.

For each shared instance on the Business CDN it request a Shared Instance on the technical CDN

TODO

TODO Before release

  • Garbage collect of stopped or destroyed slave???
  • Check domain name via CDN via software PY
  • Do not request to CDN with default parameters
  • Add clear parameters for computer uid / instance uid for sla
  • Add Egg tests
  • Use JSON parameters
  • Add subroutine to check domain validation. / Non Slapos dependent checks.
  • Request 1000 share instance to test scalability and garbage collection
  • Add promise for failing instance to help operator debug
  • Clean up connection parameters and parameters for slaves sent and received from master
  • Add SR tests

TODO after release (to be validated)

  • slapconfiguration jsonschema slaves: Master doesn't send the SR of the slave. How do you check schema.
  • How to create ticket to inform user its instance is failing
  • Resiliency of databases?? What happens if the list of validated domain is lost?

CDN Requester: Changes Since nxd/master

Executive Summary

This document presents a comprehensive overview of all changes introduced in the cdn-requester branch since nxd/master. The branch represents a major feature addition implementing a complete CDN request management system with instance node management, domain validation, and automated instance lifecycle management.

Branch: cdn-requester
Base: nxd/master
Total Commits: 62
Date Range: September 24, 2025 - December 8, 2025
Author: Cédric Le Ninivin

Statistics

Code Changes

  • Files Added: 15
  • Files Modified: 5
  • Total Lines Added: ~8,150
  • Total Lines Removed: ~10

New Components

  • New Recipes: 3 major recipes
    • cdnrequest (721 lines)
    • instancenode (936 lines)
    • localinstancedb (415 lines)
  • New Test Suites: 3 comprehensive test files
    • test_cdnrequest.py (1,697 lines)
    • test_instancenode.py (2,488 lines)
    • test_localinstancedb.py (760 lines)
  • New Software Configuration: Complete CDN requester software Release
  • New JSON Schemas: 4 schema files for validation

Key Improvements and Features

1. Domain Validation System

  • Token-based validation: Secure domain ownership verification
  • DNS resolution: Real-time DNS checking with persistent resolver
  • Uniqueness enforcement: Prevents domain conflicts across instances
  • Host tracking: Comprehensive host-to-instance mapping

2. Instance Lifecycle Management

  • Automated provisioning: Request new instances based on database comparison
  • Update detection: Hash-based change detection for efficient updates
  • Cleanup: Automatic destruction of invalid or removed instances
  • Validation pipeline: Multi-stage validation ensures instance quality

3. Standalone Execution

  • CLI support: Recipes can run independently of buildout
  • Cron integration: Automated periodic execution
  • Configuration flexibility: Flat config files and command-line options
  • Logging: Comprehensive logging with rotation support

4. Database Efficiency

  • WAL mode: Better concurrency for read/write operations
  • Hash-based comparison: Fast change detection without full comparison
  • Transaction support: Safe bulk operations
  • Schema management: Automatic schema creation and migration

5. Testing Coverage

  • Comprehensive tests: Over 4,900 lines of test code
  • Full scenarios: End-to-end install and update tests
  • Edge cases: Error handling and boundary condition tests
  • Mock cleanup: Proper test isolation

6. Monitoring and Observability

  • Progress tracking: Detailed logging of instance processing
  • Instance counting: Total instance reporting
  • Promise checks: Health monitoring for CDN requester
  • Reprocessing logs: Clear visibility into instance state changes

Major Components

1. CDN Request Recipe (slapos/recipe/cdnrequest.py)

A comprehensive recipe for managing CDN instance requests with domain validation, DNS resolution, and instance lifecycle management.

Key Features:

  • Domain Validation System

    • Custom domain validation with token-based verification
    • DNS resolution validation using persistent DNS resolver
    • Domain uniqueness checking across instances
    • Host tracking and management

  • DNS Resolution

    • Persistent DNS resolver with fresh cache at initialization
    • Direct DNS calls for validation
    • Efficient DNS validation using dnspython library

  • Instance Management

    • Request new CDN instances
    • Update existing instances
    • Destroy invalid instances
    • Parameter verification (matching rapid-cdn constraints)

  • Security Features

    • Cipher suite validation and translation
    • SSL/TLS configuration management
    • Secure token generation for domain validation

  • Standalone Execution

    • Can run as a standalone script via cdnrequest-script console command
    • Command-line argument parsing
    • Cron-based execution support

Database Schema:

  • domain_validation table: Tracks domain validation status per instance
  • used_hosts table: Tracks host assignments to instances

2. Instance Node Recipe (slapos/recipe/instancenode.py)

A recipe for managing instance nodes that compare instance databases and request instances accordingly.

Key Features:

  • Instance Comparison

    • Compares instances from instance-db-path (update_list) with stored instances
    • Uses InstanceListComparator for efficient comparison
    • Tracks valid and invalid instances

  • Instance Lifecycle Management

    • Requests new instances (if valid)
    • Updates modified instances (if valid)
    • Destroys instances no longer in the update list
    • Only publishes if parameters have changed

  • Validation Logic

    • Pre-deployment instance validation
    • Connection parameter validation
    • Instance validity checking (only valid if request is successful)
    • Instance must have connection parameters to be considered valid

  • Request Management

    • Requests made with prefix for instance reference
    • Supports shared instances
    • Software type and URL configuration

  • Logging and Monitoring

    • Comprehensive logging configuration
    • Progress tracking
    • Total instance count reporting
    • Reprocessing logs

  • Standalone Execution

    • Command-line interface
    • Configurable logging (logfile, debug mode)
    • Can run outside of buildout context

3. Local Instance Database (slapos/recipe/localinstancedb.py)

A SQLite-based database accessor for managing local instance data.

Key Features:

  • Database Management

    • SQLite database with WAL (Write-Ahead Logging) mode support
    • Automatic schema creation and migration
    • Connection pooling and timeout handling

  • Instance Storage

    • HostedInstanceLocalDB: Stores hosted instances with validation status
    • Instance parameter hashing for change detection
    • Valid and invalid instance lists

  • Comparison Logic

    • InstanceListComparator: Efficient comparison of instance lists
    • Hash-based change detection
    • Prevents database removal when lists are empty (bug prevention)

  • Database Operations

    • Bulk insert operations
    • Transaction support
    • Row factory for easy data access

4. Enhanced SlapConfiguration Recipe

Extended the existing slapconfiguration recipe with new functionality:

  • New Entry Point: slapconfiguration.jsonschema.localdb
    • JSON schema validation with local database support
    • Enables persistent storage of configuration data

Software Release: cdn-requester

A complete SlapOS software release for CDN request management.

Configuration Files

  1. software.cfg - Main software definition
  2. instance.cfg.in - Instance template for requester
  3. instancenode.cfg.in - Instance node configuration template
  4. instance-common.cfg.in - Common instance configuration
  5. buildout.hash.cfg - Buildout hash configuration

JSON Schemas

  1. instance-slave-input-schema.json (381 lines)

    • Comprehensive validation for slave instance parameters
    • Custom domain validation patterns
    • Backend URL, type, path configuration
    • Cache, SSL, and security settings
    • Enhanced validation patterns and constraints

  2. instance-slave-output-schema.json (55 lines)

    • Output schema for slave instances

  3. instance-requester-input-schema.json (38 lines)

    • Input schema for requester instances

  4. instance-requester-output-schema.json (7 lines)

    • Output schema for requester instances

Key Features:

  • Cron-based Execution

    • CDN requester runs via cron.d
    • Configurable frequency (default: 1 minute)
    • Log rotation support

  • Logging Infrastructure

    • Comprehensive logging configuration
    • Log file management
    • Log rotation setup
    • Directory cleanup

  • Monitoring

    • Promise to check if CDN requester is running
    • Process tracking
    • Progress reporting

  • Default Values

    • Default empty values for SLA parameters
    • Flat configuration for instancenode

Testing Infrastructure

Test Coverage

1. test_cdnrequest.py (1,697 lines)

Comprehensive test suite for CDN request functionality:

  • Domain validation tests
  • DNS resolution tests
  • Instance management tests
  • Full install and update scenarios
  • Error handling tests
  • Mock database cleanup

2. test_instancenode.py (2,488 lines)

Extensive test suite for instance node functionality:

  • Instance comparison tests
  • Request management tests
  • Validation logic tests
  • Command-line interface tests
  • Logging configuration tests
  • Error handling and edge cases

3. test_localinstancedb.py (760 lines)

Test suite for local database functionality:

  • Database creation and schema tests
  • Instance storage and retrieval tests
  • Comparison logic tests
  • Hash computation tests
  • Edge case handling

4. Enhanced test_slaposconfiguration.py

Updated tests for the new slapconfiguration.jsonschema.localdb functionality.

Test Organization

  • All test files moved to slapos/test/recipe/ directory
  • Added __init__.py for proper test suite integration
  • Tests ensure proper cleanup of mock database patches

Dependencies

New Dependencies Added to setup.py:

  • dnspython - For DNS resolution and validation
  • Already existing dependencies utilized:
    • jsonschema - For JSON schema validation
    • netaddr - For IP address manipulation

Console Scripts:

  • cdnrequest-script - Standalone CDN request management script

Architecture Changes

1. Recipe Entry Points

New buildout recipe entry points:

  • cdnrequest = slapos.recipe.cdnrequest:CDNRequestRecipe
  • instancenode = slapos.recipe.instancenode:Recipe
  • slapconfiguration.jsonschema.localdb = slapos.recipe.slapconfiguration:JsonSchemaWithDB

2. Execution Model

Before: Recipes executed only within buildout context

After:

  • Instance Node Recipes can run as standalone scripts
  • Command-line interface support
  • Cron-based execution
  • Independent of buildout for requests

3. Database Architecture

New: SQLite-based local instance database

  • Persistent storage of instance state
  • Efficient comparison using hashing
  • WAL mode for better concurrency
  • Schema-based data management

4. Validation Flow

New Multi-Stage Validation:

  1. Pre-deployment validation (preDeployInstanceValidation)
  2. Domain validation with DNS resolution
  3. Connection parameter validation
  4. Instance request validation
  5. Post-request validation

Configuration Examples

CDN Request Recipe Usage

[cdn-request]
recipe = cdnrequest
instance-db-path = ${buildout:directory}/var/instance-db.sqlite
requestinstance-db-path = ${buildout:directory}/var/request-db.sqlite
server-url = https://slapos.example.com
computer-id = COMP-123
partition-id = PART-456
software-url = https://software.example.com/cdn-requester
software-type = cdn-requester
request-name-prefix = cdn-

Instance Node Recipe Usage

[instance-node]
recipe = instancenode
instance-db-path = ${buildout:directory}/var/instance-db.sqlite
requestinstance-db-path = ${buildout:directory}/var/request-db.sqlite
server-url = https://slapos.example.com
computer-id = COMP-123
partition-id = PART-456
software-url = https://software.example.com/cdn-requester
software-type = cdn-requester
logfile = ${buildout:directory}/var/log/instancenode.log
debug = false

Standalone Execution

# Run CDN request script
cdnrequest-script --config /path/to/config.ini --logfile /path/to/log.log

# Run instance node
python -m slapos.recipe.instancenode --config /path/to/config.ini

Migration Notes

For Existing Deployments

  1. New Dependencies Required:

    • dnspython must be installed
    • Update setup.py or install manually

  2. Database Migration:

    • New SQLite databases will be created automatically
    • No manual migration needed for new deployments

  3. Configuration Changes:

    • New recipe entry points available
    • JSON schemas must be updated
    • Cron configuration may be needed for automated execution

  4. Testing:

    • All tests moved to slapos/test/recipe/
    • Run test suite to verify compatibility

Future Considerations

Potential Enhancements

  1. Performance Optimization:

    • Batch DNS resolution
    • Parallel instance processing
    • Caching improvements

  2. Monitoring:

    • Metrics collection
    • Alerting integration
    • Dashboard support

  3. Scalability:

    • Multi-node support
    • Load balancing
    • Distributed database options

  4. Security:

    • Enhanced token management
    • Rate limiting
    • Audit logging

Conclusion

The cdn-requester branch represents a significant addition to the SlapOS cookbook, introducing a complete CDN request management system with:

  • 3 major new recipes with over 2,000 lines of production code
  • Comprehensive test coverage with over 4,900 lines of tests
  • Complete software package with schemas and configurations
  • Standalone execution capabilities
  • Robust domain validation and DNS resolution
  • Efficient instance lifecycle management

The implementation follows best practices with:

  • Proper error handling
  • Comprehensive logging
  • Extensive testing
  • Clear separation of concerns
  • Modular architecture

All changes are backward compatible and can be integrated into the main branch with appropriate testing and validation.

Appendix: File Changes Summary

New Files

slapos/recipe/cdnrequest.py (721 lines)
slapos/recipe/instancenode.py (936 lines)
slapos/recipe/localinstancedb.py (415 lines)
slapos/test/recipe/__init__.py
slapos/test/recipe/test_cdnrequest.py (1,697 lines)
slapos/test/recipe/test_instancenode.py (2,488 lines)
slapos/test/recipe/test_localinstancedb.py (760 lines)
software/cdn-requester/buildout.hash.cfg
software/cdn-requester/instance-common.cfg.in
software/cdn-requester/instance-requester-input-schema.json
software/cdn-requester/instance-requester-output-schema.json
software/cdn-requester/instance-slave-input-schema.json (381 lines)
software/cdn-requester/instance-slave-output-schema.json
software/cdn-requester/instance.cfg.in
software/cdn-requester/instancenode.cfg.in
software/cdn-requester/software.cfg
software/cdn-requester/software.cfg.json

Modified Files

setup.py (added dnspython dependency, new entry points)
slapos/recipe/slapconfiguration.py (added jsonschema.localdb)
slapos/test/recipe/test_slaposconfiguration.py (updated tests)
software/rapid-cdn/instance-slave-input-schema.json (enhanced validation)

Detailed Commit History by Theme

Phase 1: Foundation (September-October 2025)

Initial Development

  • 97ff80de (2025-09-24): cdn-requester: WIP
  • dc87592b (2025-09-25): WIP
  • 5984a21a (2025-10-06): wip: requesting instances
  • 967c1613 (2025-10-06): wip: publish slave
  • dc8d9d56 (2025-10-06): WIP: Remove currently uneeded files
  • 0c81d3d4 (2025-10-15): wip: Move to slapconfiguration.jsonschema

Focus: Initial exploration and architecture design

Phase 2: Core Implementation (November 2025)

Database and Configuration Foundation

  • e5c09612 (2025-11-07): slapconfiguration: introduce slapconfiguration .jsonschema.localdb recipe

    • Added local database support to slapconfiguration

  • 7abc4cc1 (2025-11-14): reciperequestinstancelist: add recipe for instance management

    • Initial instance management recipe

Domain Validation

  • da200b3b (2025-11-19): cdnrequest: add custom domain validation for CDN requests
  • b1566bdc (2025-11-19): Update instance-slave-input-schema.json to enhance validation patterns and add constraints

Enhanced Instance Management

  • b1e148d0 (2025-11-20): cdnrequest: enhance domain validation logic and instance management
  • d0c9db61 (2025-11-20): test_cdnrequest: add full scenario for install and update
  • 7744d5a5 (2025-11-20): cdnrequest: Add parameter verifications done in rapid-cdn
  • 83f7c843 (2025-11-20): cdnrequest: add constraints on server alias

Database Refactoring

  • 29c38f30 (2025-11-20): localinstancedb: Rename and cleanup

Connection Parameter Publishing

  • 4cff8c4e (2025-11-21): requestinstancelist: publish connection parameters to master
  • 7837c61c (2025-11-21): cdnrequest: Improve error message for requester
  • 7eaf8620 (2025-11-24): requestinstancelist: Update instance handling to include error information

DNS and Caching

  • 435638cb (2025-11-26): recipe.cdnrequest: Add cache and direct dns call
  • b6d28b5e (2025-11-26): recipe.cdnrequest: Fix destroyed invalid instance logic and add tests

Schema and Setup

  • 1dd96c58 (2025-11-26): cdn-requester: fix schemas
  • e1960f43 (2025-11-26): setup.py: Fix recipe entries and add dependency to dnspython
  • 92a1f9ef (2025-11-26): cdn-requester: Move to use cdnrequest recipe

Parameter Defaults

  • 686d6515 (2025-11-28): cdnrequester: Only set parameter defaults for main instance

Phase 3: Standalone Execution and Refactoring (December 2025)

Standalone Script Support

  • 9ae2c461 (2025-12-03): WIP cdnrequester: try to make it work in command line
  • 00158f2f (2025-12-03): requestinstancelist: add tests on calling the code as command line
  • 8c6edcc5 (2025-12-03): cdnrequest: add tests on dns resolution
  • 811b251a (2025-12-03): cdn-requester: run cdn requester as a standalone script

DNS Resolver Enhancement

  • 734c2bac (2025-12-03): CDNRequestRecipe: Implement a persistent DNS resolver with fresh cache at initialization to improve DNS validation efficiency.

Database Improvements

  • c5a5397f (2025-12-03): localinstancedb: prevent removal of database when valid and invalid list are empty (assume it is a bug)

Renaming and Refactoring

  • af147ac4 (2025-12-03): instancenode: Rename as instancenode to reflect behaviour
  • 9dd70c1e (2025-12-03): cdnrequester: instancenoce.cfg has flat config
  • 02738a28 (2025-12-03): cdn-requester: set default empty values for sla parameters

Buildout Independence

  • 772a7eea (2025-12-03): instancenode: move out of buildout for requests
  • 85331152 (2025-12-03): instancenode: Do not mark an instance as valid if no parameters is returned by the request

Cron and Logging

  • dce9f552 (2025-12-04): cdnrequest: Have cdn request run with cron.d
  • ef81a0b0 (2025-12-04): cdnrequest: add basic logging config

Optimization

  • f25b5dce (2025-12-04): localinstancedb: compute hash only from instance parameters
  • dbcbb464 (2025-12-04): instancenode: remove unused buildout section and only publish if parameters to published have changed
  • 3b309d3d (2025-12-04): cdnrequest: remove useless calls to buildout
  • 2a1b7112 (2025-12-04): instancenode: remove unused function
  • 8eecc33b (2025-12-04): dnsrequest: remove unused variable

Validation Improvements

  • d2039b41 (2025-12-04): instancenode: fixup instance is only valid once the request is successful
  • 95fe7704 (2025-12-04): instancenode: consider an instance is valid only if it has connection parameters
  • dc137f08 (2025-12-04): instancenode: requests are made with a prefix for isntance reference

Test Updates

  • 06ce7a22 (2025-12-04): test_instancenode: Update test to match latest changes

Major Refactoring

  • caca4ace (2025-12-05): instancenode: refactor instance processing to clarify validation and deployment flow
  • 02e86918 (2025-12-05): cdnrequest: update tests
  • b2fb938c (2025-12-05): instancenode: rename validateInstance in preDeployInstanceValidation
  • e278adca (2025-12-05): test.recipe: Add missing __init__ file
  • 07d526b5 (2025-12-05): instancenode: refactor logging configuration and enhance command-line argument parsing
  • a10ff5c3 (2025-12-05): test.cdnrequest: ensure cleanup of mock database patch in tests

Final Polish (December 8, 2025)

  • 90927672 (2025-12-08): test.localinstancedb: Move to test recipe folder to be run by Test suite
  • 572268f7 (2025-12-08): cdn-requester: Add log, logrotate and cleanup directories
  • 9eda4b48 (2025-12-08): recipe.instancenode: Add log to track process
  • 62de0e53 (2025-12-08): cdn-requester: fixup logfile and move to one minute frequency
  • d04e1106 (2025-12-08): cdn-requester: add promise to check it is running
  • d921f823 (2025-12-08): instancenode: Add more clear reprocessing log
  • ae95c1ac (2025-12-08): recipe.instancenode: add more log to track progress
  • b264dcf5 (2025-12-08): recipe.instancenade: group initial logging functions
  • d57d15b6 (2025-12-08): recipe.instancenode: report total number of instances on the instance node

Document Generated: December 2025
Branch: cdn-requester
Base: nxd/master

Edited by Cédric Le Ninivin
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7