Operations

This is the day-2 runbook for a production Warka deployment: how to back up and restore, how to apply updates safely, what to monitor, where the logs are, how to diagnose the common failures, and how the system scales on a small server. It assumes the single-VPS topology from Installation and Deployment. Commands use the placeholders your-site.example.com and $SITE; substitute your real site name. Deployment-specific paths and service names follow the deploy kit's conventions.

Service layout

supervisorctl status                     # gunicorn / workers / scheduler / redis
systemctl status cyber-zeb-frontend      # Next.js frontend
systemctl status caddy                   # reverse proxy and TLS

Backups

Warka's data lives in MariaDB plus the site's files. Use bench's backup, which captures both:

# Database + public and private files
bench --site your-site.example.com backup --with-files

Backups land under the site's backup directory, frappe-bench/sites/your-site.example.com/private/backups/, as a timestamped SQL dump plus tar archives of the files. The deploy kit includes a backup script suitable for a nightly cron entry; schedule it and copy the archives off the box so a lost server is not a lost company.

Restoring

bench --site your-site.example.com restore /path/to/backup.sql.gz \
  --with-public-files /path/to/files.tar \
  --with-private-files /path/to/private-files.tar
bench --site your-site.example.com migrate

Run migrate after a restore so the schema and the self-heal defaults are brought current. Test a restore periodically; an untested backup is a hope, not a backup.

Updating

An update ships new code and re-converges the site. The deploy kit's update script performs these steps; run it rather than the steps by hand where possible.

1. Pull the new code onto the server.
2. Migrate the backend: bench --site your-site.example.com migrate. This applies schema changes and, critically, runs the after_migrate hooks, which re-apply the custom fields, lifecycle templates, and operational defaults. The self-heal runs on every migrate, so a missing or deleted master is restored automatically. See Extending and Customizing for the pattern.
3. Rebuild the frontend: next build, then restart the frontend service.
4. Restart services:

supervisorctl restart all                # backend (gunicorn, workers, scheduler)
systemctl restart cyber-zeb-frontend     # frontend
systemctl reload caddy                   # only after editing the Caddyfile

Updates preserve data. Always take a backup before a significant update.

Monitoring and health

• A fresh site needs the setup wizard once. The deploy runs complete_setup_wizard from the company config on first provision. If you create a site by hand, run it once (see Configuration and Customization); it is a no-op afterward.
• HTTPS health: the browser padlock plus journalctl -u caddy confirm the certificate issued and renews.
• Backend health: supervisorctl status should show every process RUNNING. The dashboards themselves are a good end-to-end signal; if get_business_dashboard returns data, the full path (Caddy to Next.js to gunicorn to MariaDB and Redis) is working.

Log locations

The whitelisted API logs server-side detail (payroll failures, report errors) to the Error Log via frappe.log_error, while returning a sanitized message to the client, so check the Error Log when a user reports a vague failure.

Common issues

Payroll for large companies

run_payroll reuses HRMS logic, which offloads slip creation and submission to background workers above 30 employees to avoid a request timeout. For those runs the endpoint returns {queued: true, payroll_entry, ...} immediately. The client (and an operator debugging it) drives it to completion by polling get_payroll_status until creation_complete, calling submit_payroll_slips, then polling until complete. If a run appears stuck, confirm the background workers are RUNNING under supervisor; a stopped worker is the usual cause. The full contract is in the API Reference.

Scaling notes

Warka is designed to run comfortably on a small VPS:

• Redis-cached dashboards. The company-wide overviews (get_hr_overview, get_finance_overview, get_business_dashboard) are computed at most once per 60-second window per company and served from Redis, so many concurrent users cost one database computation, not one per user. Writes that change the numbers bust the cache, so data stays fresh. See Architecture.
• Background workers. Long tasks (large payroll runs, and any enqueued job) execute on the supervisor-managed workers, keeping web requests fast. Adding worker capacity is the first lever when payroll for a very large workforce is slow.
• Everything on localhost. The database, cache, backend, and frontend all talk over localhost, so the only network hop is the browser to Caddy. Vertical scaling (more RAM and vCPU on the one box) is the simplest growth path; MariaDB, Redis, and the app can later be split onto separate hosts if a single company genuinely outgrows one server.

For the rollout and go-live checklist (backups confirmed, HTTPS working, a pilot transaction per team), see the Implementation Guide.