snapshot of https://github.com/ActivityWatch/aw-server-rust.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
Build / ubuntu-latest (push) Failing after 5s
Build / Android (push) Failing after 2s
Build / Code coverage (push) Failing after 2s
Lint / format (push) Failing after 3s
Lint / clippy (push) Failing after 3s
Build / macOS-latest (push) Has been cancelled
Build / windows-latest (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-16 00:19:17 +05:00
commit da934b693c
120 changed files with 26659 additions and 0 deletions

44
aw-sync/Cargo.toml Normal file
View File

@@ -0,0 +1,44 @@
[package]
name = "aw-sync"
version = "0.1.0"
authors = ["Erik Bjäreholt <erik@bjareho.lt>"]
edition = "2021"
[lib]
name = "aw_sync"
path = "src/lib.rs"
crate-type = ["cdylib", "rlib"]
[[bin]]
name = "aw-sync"
path = "src/main.rs"
[dependencies]
log = "0.4"
toml = "0.8"
chrono = { version = "0.4", features = ["serde"] }
serde = "1.0"
serde_json = "1.0"
reqwest = { version = "0.12", features = ["json", "blocking"] }
dirs = "6"
gethostname = "0.4.3"
# CLI-only dependencies (optional)
clap = { version = "4.1", features = ["derive"], optional = true }
ctrlc = { version = "3.4.5", optional = true }
aw-server = { path = "../aw-server" }
aw-models = { path = "../aw-models" }
aw-datastore = { path = "../aw-datastore" }
aw-client-rust = { path = "../aw-client-rust" }
[target.'cfg(target_os="linux")'.dependencies]
openssl = { version = "0.10.80", features = ["vendored"] } # https://github.com/ActivityWatch/aw-server-rust/issues/478
[target.'cfg(target_os="android")'.dependencies]
jni = { version = "0.21", default-features = false }
android_logger = "0.13"
[features]
default = ["cli"]
cli = ["clap", "ctrlc"]

142
aw-sync/README.md Normal file
View File

@@ -0,0 +1,142 @@
aw-sync-rust
============
Synchronization for ActivityWatch.
Works by syncing local buckets with a special folder, which in turn should be synchronized by rsync/Syncthing/Dropbox/GDrive/etc.
The latest beta versions of ActivityWatch ship with the `aw-sync` binary, but it's not enabled by default. You can start it from aw-qt or the command line, but due to the early state of development might not have the best UX. Please report issues and submit PRs!
Was originally prototyped as a PR to aw-server: https://github.com/ActivityWatch/aw-server/pull/50
## Usage
This will start a daemon which pulls and pushes events with the sync directory (`~/ActivityWatchSync` by default) every 5 minutes:
```sh
# Basic sync daemon (syncs all buckets every 5 minutes)
aw-sync
# Same as above
aw-sync daemon
# Sync daemon with specific buckets only
aw-sync daemon --buckets "aw-watcher-window,aw-watcher-afk" --start-date "2024-01-01"
# Sync daemon in push-only or pull-only mode
aw-sync daemon --mode push
aw-sync daemon --mode pull
# Sync all buckets once and exit
aw-sync sync --start-date "2024-01-01"
```
For more options, see `aw-sync --help`. Some notable options:
- `--config`: Read the local server port and API key from a custom aw-server config file. Pass the same path to both aw-server and aw-sync when using aw-server's `--config` override.
- `--buckets`: Specify which buckets to sync (comma-separated). By default, all buckets are synced.
- Use `--buckets "bucket1,bucket2"` to sync specific buckets
- Not specifying this option syncs all buckets by default
- `--start-date`: Only sync events after this date (YYYY-MM-DD)
- `--sync-db`: Specify a specific database file in the sync directory
- `--mode`: Choose sync mode: "push", "pull", or "both" (default: "both")
- For `aw-sync sync`, passing `--mode` (with no other options) is enough to opt into the per-bucket sync path where it's respected; without it, or any of `--buckets`/`--start-date`/`--sync-db`, `aw-sync sync` falls back to the legacy host-based mode, which always does both a pull and a push
### Setting up sync
Once you have aw-sync running, you need to set up syncing with the sync directory using your preferred syncing tool.
The default sync directory is `~/ActivityWatchSync`, but you can change it using the `--sync-dir` option or by setting the `AW_SYNC_DIR` environment variable.
### Running from source
If you want to run it from source, in the root of the repository run:
```sh
cargo run --bin aw-sync
```
For more options, see `cargo run --bin aw-sync -- --help`.
## FAQ
### When is it ready?
It works today, but there is still testing and polishing to be done before it's "click and play".
### Why do it this way?
By essentially only offering a "sync with folder" feature, we give the user a lot of freedom to choose how to store and sync their data.
We also avoid having to implement complex features such as conflict resolution, by enforcing that each device only writes to files in the sync folder they "own", and other devices may not modify them.
### What are the limitations?
- It only syncs afk and window buckets by default (since bucket IDs need to be unique)
- It will work a lot better once proper `hostname -> device ID` migration is complete.
- It doesn't sync settings
- It doesn't support Android, yet.
- It mirrors events to all devices,
- If you have a lot of devices you'll get a lot of duplicates, taking up a lot of space and potentially impacting performance.
- It doesn't support modifying/deleting events, yet.
---
## Advanced usage
### Syncing real data to a testing instance
If you want to try sync, you can do so by following these steps.
We will use a separate testing instance of aw-server(-rust) to store and view the synced data from the sync directory. This is to avoid testing against & potentially pollute production instances in write-scenarios. We will sync all devices with the sync folder, and then sync the sync folder into our testing instance to view.
To test syncing real events to a sync folder which can then be pulled from.
We will use some helper scripts to do the following:
1. `./test-sync-push.sh`
- Creates a sync directory **for you to set up sync** with Syncthing/Dropbox/Gdrive/rclone/whatever
- By default `~/ActivityWatchSync`
- Creates a datastore for the current host in the sync folder
- Sync all local buckets of interest (window & afk buckets, by default) to the sync dir
2. `./test-server.sh`
- Starts a testing server **on port 5667** using a temporary directory as datastore (`/tmp/...`)
3. `./test-sync-pull.sh`
- Imports all the events from sync folder into the testing server on port 5667
4. You should now have all events synced to a local testing instance!
- You can browse [127.0.0.1:5667](http://127.0.0.1:5667) to view testing instance, where you'll see events from synced all hosts.
- You can now set up syncing for `~/ActivityWatchSync` on more devices, and on each one use the script `./test-sync.sh` to push their events into the sync folder, then run `./test-import-sync.sh` on the device where you have the testing instance to update the data there.
5. To view data from all devices at once, go into [127.0.0.1:5667/#/settings](127.0.0.1:5667/#/settings) and check the "Use multidevice query" checkbox (near the bottom, under "developer settings").
- You can now navigate back to the activity view for any device, where you should see data from multiple devices being included in (most of) the visualizations.
In the end, You should get something like this: https://twitter.com/ErikBjare/status/1519399784234246147
### Testing with fake data
#### Pushing to the sync directory
First start the sending aw-server instance. For example:
```sh
PORT=5667
cargo run --bin aw-server -- --testing --port $PORT --dbpath test-$PORT.sqlite --device-id $PORT --no-legacy-import
```
You can create some test data by opening `http://localhost:5667/#/stopwatch` and creating a few events.
Then run `cargo run --bin aw-sync-rust -- --port 5667` to sync your instance's buckets with the target directory.
#### Pulling from the sync directory
Now to sync things back from the sync directory into another instance.
First, lets start another instance:
```sh
PORT=5668
cargo run --bin aw-server -- --testing --port $PORT --dbpath test-$PORT.sqlite --device-id $PORT --no-legacy-import
```
Now run `cargo run --bin aw-sync -- --port 5668` to pull buckets from the sync dir (retrieving events from the 5667 instance) and push buckets from the 5668 instance to the sync dir.

114
aw-sync/src/accessmethod.rs Normal file
View File

@@ -0,0 +1,114 @@
use std::collections::HashMap;
use aw_client_rust::blocking::AwClient;
use chrono::{DateTime, Utc};
use reqwest::StatusCode;
use aw_datastore::{Datastore, DatastoreError};
use aw_models::{Bucket, Event};
// This trait should be implemented by both AwClient and Datastore, unifying them under a single API
pub trait AccessMethod: std::fmt::Debug {
fn get_buckets(&self) -> Result<HashMap<String, Bucket>, String>;
fn get_bucket(&self, bucket_id: &str) -> Result<Bucket, DatastoreError>;
fn create_bucket(&self, bucket: &Bucket) -> Result<(), DatastoreError>;
fn get_events(
&self,
bucket_id: &str,
start: Option<DateTime<Utc>>,
end: Option<DateTime<Utc>>,
limit: Option<u64>,
) -> Result<Vec<Event>, String>;
fn insert_events(&self, bucket_id: &str, events: Vec<Event>) -> Result<(), String>;
fn get_event_count(&self, bucket_id: &str) -> Result<i64, String>;
fn heartbeat(&self, bucket_id: &str, event: Event, duration: f64) -> Result<(), String>;
}
/// Every method here returns a `Result`, so a datastore failure must be reported
/// through it rather than unwrapped.
///
/// These used to `unwrap()`, which on Android turns any datastore error (a
/// dead worker thread, a locked or corrupt database, a disappearing sync file)
/// into a panic that unwinds out of the JNI `extern "C"` frame and aborts the
/// whole app — see ActivityWatch/aw-android#220.
impl AccessMethod for Datastore {
fn get_buckets(&self) -> Result<HashMap<String, Bucket>, String> {
Datastore::get_buckets(self).map_err(|e| format!("{e:?}"))
}
fn get_bucket(&self, bucket_id: &str) -> Result<Bucket, DatastoreError> {
Datastore::get_bucket(self, bucket_id)
}
fn create_bucket(&self, bucket: &Bucket) -> Result<(), DatastoreError> {
Datastore::create_bucket(self, bucket)?;
self.force_commit()?;
Ok(())
}
fn get_events(
&self,
bucket_id: &str,
start: Option<DateTime<Utc>>,
end: Option<DateTime<Utc>>,
limit: Option<u64>,
) -> Result<Vec<Event>, String> {
Datastore::get_events(self, bucket_id, start, end, limit).map_err(|e| format!("{e:?}"))
}
fn heartbeat(&self, bucket_id: &str, event: Event, duration: f64) -> Result<(), String> {
Datastore::heartbeat(self, bucket_id, event, duration).map_err(|e| format!("{e:?}"))?;
self.force_commit().map_err(|e| format!("{e:?}"))?;
Ok(())
}
fn insert_events(&self, bucket_id: &str, events: Vec<Event>) -> Result<(), String> {
Datastore::insert_events(self, bucket_id, &events[..]).map_err(|e| format!("{e:?}"))?;
self.force_commit().map_err(|e| format!("{e:?}"))?;
Ok(())
}
fn get_event_count(&self, bucket_id: &str) -> Result<i64, String> {
Datastore::get_event_count(self, bucket_id, None, None).map_err(|e| format!("{e:?}"))
}
}
impl AccessMethod for AwClient {
fn get_buckets(&self) -> Result<HashMap<String, Bucket>, String> {
AwClient::get_buckets(self).map_err(|e| e.to_string())
}
fn get_bucket(&self, bucket_id: &str) -> Result<Bucket, DatastoreError> {
let bucket = AwClient::get_bucket(self, bucket_id);
match bucket {
Ok(bucket) => Ok(bucket),
Err(e) => {
warn!("{:?}", e);
let code = e.status().unwrap_or(StatusCode::INTERNAL_SERVER_ERROR);
if code == StatusCode::NOT_FOUND {
Err(DatastoreError::NoSuchBucket(bucket_id.into()))
} else {
Err(DatastoreError::InternalError(format!(
"Unexpected error fetching bucket {bucket_id}: {e}"
)))
}
}
}
}
fn get_events(
&self,
bucket_id: &str,
start: Option<DateTime<Utc>>,
end: Option<DateTime<Utc>>,
limit: Option<u64>,
) -> Result<Vec<Event>, String> {
AwClient::get_events(self, bucket_id, start, end, limit).map_err(|e| e.to_string())
}
fn insert_events(&self, bucket_id: &str, events: Vec<Event>) -> Result<(), String> {
AwClient::insert_events(self, bucket_id, events).map_err(|e| e.to_string())
}
fn get_event_count(&self, bucket_id: &str) -> Result<i64, String> {
AwClient::get_event_count(self, bucket_id).map_err(|e| e.to_string())
}
fn create_bucket(&self, bucket: &Bucket) -> Result<(), DatastoreError> {
AwClient::create_bucket(self, bucket)
.map_err(|e| DatastoreError::InternalError(e.to_string()))?;
Ok(())
}
fn heartbeat(&self, bucket_id: &str, event: Event, duration: f64) -> Result<(), String> {
AwClient::heartbeat(self, bucket_id, &event, duration).map_err(|e| format!("{e:?}"))
}
}

400
aw-sync/src/android.rs Normal file
View File

@@ -0,0 +1,400 @@
use std::ffi::CString;
use std::os::raw::{c_char, c_int, c_void};
use std::panic;
use std::path::Path;
use std::sync::Once;
use aw_client_rust::blocking::AwClient;
use aw_server::panic_guard::catch_panic;
use jni::objects::{JClass, JString};
use jni::sys::{jint, jstring, JNI_VERSION_1_6};
use jni::JNIEnv;
use serde_json::json;
use crate::{pull, pull_all, push_with_hostname};
const ANDROID_LOG_FATAL: c_int = 7;
const ANDROID_LOG_TAG: &str = "aw-sync";
/// logcat truncates a single `__android_log_write` payload around 4 KiB.
const MAX_LOG_BYTES: usize = 4000;
#[link(name = "log")]
extern "C" {
fn __android_log_write(prio: c_int, tag: *const c_char, text: *const c_char) -> c_int;
}
/// Write `msg` to logcat, splitting on the 4 KiB limit and stripping NULs so
/// `CString::new` cannot fail. Never panics.
fn android_log_fatal(msg: &str) {
let Ok(tag) = CString::new(ANDROID_LOG_TAG) else {
return;
};
let sanitized = msg.replace('\0', "\\0");
let mut rest = sanitized.as_str();
while !rest.is_empty() {
let mut idx = rest.len().min(MAX_LOG_BYTES);
while idx > 0 && !rest.is_char_boundary(idx) {
idx -= 1;
}
if idx == 0 {
break;
}
if let Ok(text) = CString::new(&rest[..idx]) {
unsafe {
__android_log_write(ANDROID_LOG_FATAL, tag.as_ptr(), text.as_ptr());
}
}
rest = &rest[idx..];
}
}
/// Install a panic hook that writes to logcat via `android_log`.
///
/// The default Rust hook writes to stderr, which Android discards, so a panic
/// in this `.so` currently surfaces only as `SIGABRT` with no message
/// (ActivityWatch/aw-android#220).
fn install_panic_hook() {
panic::set_hook(Box::new(|info| {
let payload = if let Some(s) = info.payload().downcast_ref::<&'static str>() {
*s
} else if let Some(s) = info.payload().downcast_ref::<String>() {
s.as_str()
} else {
"Box<dyn Any>"
};
let location = match info.location() {
Some(loc) => format!("{}:{}:{}", loc.file(), loc.line(), loc.column()),
None => "unknown location".to_string(),
};
let thread = std::thread::current();
let thread_name = thread.name().unwrap_or("<unnamed>");
let header = format!("Rust panic in thread '{thread_name}' at {location}: {payload}");
android_log_fatal(&header);
error!("{}", header);
let backtrace =
panic::catch_unwind(|| format!("{}", std::backtrace::Backtrace::force_capture()));
if let Ok(bt) = backtrace {
android_log_fatal("Backtrace:");
for line in bt.lines() {
android_log_fatal(line);
}
}
}));
}
/// Route `log` macros to logcat and install the panic hook. Safe to call more than once.
fn init_android_logging() {
static INIT: Once = Once::new();
INIT.call_once(|| {
android_logger::init_once(
android_logger::Config::default()
.with_max_level(log::LevelFilter::Info)
.with_tag(ANDROID_LOG_TAG),
);
install_panic_hook();
info!("aw-sync: android_logger and panic hook installed");
});
}
/// Called automatically when `System.loadLibrary("aw_sync")` loads this `.so`,
/// so panics are diagnosable even if a JNI entry point is never reached.
#[no_mangle]
pub extern "system" fn JNI_OnLoad(_vm: *mut jni::sys::JavaVM, _reserved: *mut c_void) -> jint {
// Guarded like the other entry points: this is the one that runs before any
// logging exists, so an unwind out of it would abort the app during
// `System.loadLibrary("aw_sync")` with nothing in logcat at all.
let _ = catch_panic("JNI_OnLoad", init_android_logging);
JNI_VERSION_1_6
}
/// Helper function to convert Rust string to Java string.
///
/// Must not panic: it is also used on the guard's error path, where a second
/// unwind would reach the `extern "C"` frame and abort after all. A null return
/// surfaces in Kotlin as a `NullPointerException` at the call site, which
/// `SyncInterface.performSyncAsync` already catches — recoverable, unlike
/// `SIGABRT`.
fn rust_string_to_jstring(env: &JNIEnv, s: String) -> jstring {
match env.new_string(s) {
Ok(output) => output.into_raw(),
Err(e) => {
error!("Couldn't create java string: {}", e);
std::ptr::null_mut()
}
}
}
/// The response shape `SyncInterface.performSyncAsync` parses: it reads
/// `success` and then either `message` or `error`.
fn sync_error_json(msg: &str) -> String {
json!({
"success": false,
"error": msg
})
.to_string()
}
/// Run a `jstring`-returning JNI entry point with panics caught.
///
/// A panic that unwinds out of an `extern "C"` function aborts the process
/// (Rust >= 1.81), which is how a plain `unwrap()` on the sync path takes the
/// whole app down — see ActivityWatch/aw-android#220 and #267. Catching it here
/// turns that abort into the ordinary `{"success": false, "error": ...}` object
/// the Kotlin caller already handles.
///
/// The panic hook installed by `init_android_logging` still runs first, so the
/// message and backtrace reach logcat before the unwind is stopped.
fn jni_guard<F>(env: &mut JNIEnv, name: &str, f: F) -> jstring
where
F: FnOnce(&mut JNIEnv) -> jstring,
{
match catch_panic(name, || {
init_android_logging();
f(&mut *env)
}) {
Ok(result) => result,
Err(msg) => {
error!("{}", msg);
android_log_fatal(&msg);
// A panic in the middle of a JNI call can leave a pending Java
// exception, which would make the following NewStringUTF fail.
let _ = env.exception_clear();
rust_string_to_jstring(env, sync_error_json(&msg))
}
}
}
/// Run a `void` JNI entry point with panics caught. There is no return value to
/// carry an error, so the panic is logged and swallowed.
fn jni_guard_void<F>(name: &str, f: F)
where
F: FnOnce(),
{
if let Err(msg) = catch_panic(name, || {
init_android_logging();
f();
}) {
error!("{}", msg);
android_log_fatal(&msg);
}
}
/// Point this library's `ANDROID_DATA_DIR` at the app filesDir.
///
/// `libaw_sync.so` and `libaw_server.so` are separate cdylibs, so
/// `RustInterface.setDataDir` does not update the copy compiled into this
/// `.so`. SyncInterface.kt already sets `XDG_DATA_HOME=$filesDir/data`, which
/// is the path that works for debug (`applicationIdSuffix ".debug"`) and
/// work-profile installs — the hardcoded default is only the release user-0
/// path.
fn apply_android_data_dir_from_env() {
let Ok(xdg_data) = std::env::var("XDG_DATA_HOME") else {
return;
};
let Some(files_dir) = crate::dirs::files_dir_from_xdg_data_home(Path::new(&xdg_data)) else {
warn!(
"XDG_DATA_HOME={} is not $filesDir/data; leaving android data dir unchanged",
xdg_data
);
return;
};
let path = files_dir.to_string_lossy();
info!("android data dir from XDG_DATA_HOME: {}", path);
aw_server::dirs::set_android_data_dir(&path);
}
/// Mirror of `RustInterface.setDataDir`. Prefer this explicit path; the XDG
/// fallback in `get_client` covers current Kotlin that does not call it.
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_setDataDir(
mut env: JNIEnv,
_class: JClass,
java_dir: JString,
) {
jni_guard_void("setDataDir", || match env.get_string(&java_dir) {
Ok(s) => {
let path: String = s.into();
info!("Setting android data dir as {}", path);
aw_server::dirs::set_android_data_dir(&path);
}
Err(e) => {
error!("setDataDir: failed to read path: {}", e);
}
});
}
/// Helper function to get AwClient from port.
///
/// Android enables API-key auth whenever `config.toml` has `[auth].api_key`.
/// The desktop CLI path (`main.rs`) already forwards that key; this JNI path
/// used `AwClient::new()` and 401'd on `GET /api/0/buckets` (aw-android#247).
fn get_client(port: i32) -> Result<AwClient, String> {
apply_android_data_dir_from_env();
let host = "127.0.0.1";
let api_key = match crate::util::get_server_config(false, None) {
Ok(cfg) => {
if cfg.api_key.is_some() {
info!("using API key from config.toml for local client");
}
cfg.api_key
}
Err(e) => {
warn!("failed to read server config for API key: {}", e);
None
}
};
AwClient::new_with_api_key(host, port as u16, "aw-sync-android", api_key)
.map_err(|e| format!("Failed to create client: {}", e))
}
/// Render a sync result as the JSON object the Kotlin side parses.
fn sync_result_to_jstring(env: &JNIEnv, name: &str, result: Result<String, String>) -> jstring {
match result {
Ok(msg) => rust_string_to_jstring(env, msg),
Err(e) => {
error!("{} error: {}", name, e);
rust_string_to_jstring(env, sync_error_json(&e))
}
}
}
/// Pull sync data from all hosts in the sync directory
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_syncPullAll(
mut env: JNIEnv,
_class: JClass,
port: i32,
// syncPullAll iterates every host found in the sync directory, so the
// hostname the Kotlin side passes is unused. Kept for signature parity with
// the other SyncInterface natives.
_hostname: JString,
) -> jstring {
jni_guard(&mut env, "syncPullAll", |env| {
let result: Result<String, String> = (|| {
let client = get_client(port)?;
pull_all(&client).map_err(|e| format!("Sync pull failed: {}", e))?;
Ok(json!({
"success": true,
"message": "Successfully pulled from all hosts"
})
.to_string())
})();
sync_result_to_jstring(env, "syncPullAll", result)
})
}
/// Pull sync data from a specific host
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_syncPull(
mut env: JNIEnv,
_class: JClass,
port: i32,
hostname: JString,
) -> jstring {
jni_guard(&mut env, "syncPull", |env| {
let result: Result<String, String> = (|| {
let client = get_client(port)?;
let hostname_str: String = env
.get_string(&hostname)
.map_err(|e| format!("Failed to get hostname string: {}", e))?
.into();
pull(&hostname_str, &client).map_err(|e| format!("Sync pull failed: {}", e))?;
Ok(json!({
"success": true,
"message": format!("Successfully pulled from host: {}", hostname_str)
})
.to_string())
})();
sync_result_to_jstring(env, "syncPull", result)
})
}
/// Push local sync data to the sync directory
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_syncPush(
mut env: JNIEnv,
_class: JClass,
port: i32,
hostname: JString,
) -> jstring {
jni_guard(&mut env, "syncPush", |env| {
let result: Result<String, String> = (|| {
let hostname_str: String = env
.get_string(&hostname)
.map_err(|e| format!("Failed to get hostname: {}", e))?
.into();
let client = get_client(port)?;
push_with_hostname(&client, &hostname_str)
.map_err(|e| format!("Sync push failed: {}", e))?;
Ok(json!({
"success": true,
"message": "Successfully pushed local data"
})
.to_string())
})();
sync_result_to_jstring(env, "syncPush", result)
})
}
/// Perform full sync (pull from all hosts, then push local data)
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_syncBoth(
mut env: JNIEnv,
_class: JClass,
port: i32,
hostname: JString,
) -> jstring {
jni_guard(&mut env, "syncBoth", |env| {
let result: Result<String, String> = (|| {
let hostname_str: String = env
.get_string(&hostname)
.map_err(|e| format!("Failed to get hostname: {}", e))?
.into();
let client = get_client(port)?;
pull_all(&client).map_err(|e| format!("Pull phase failed: {}", e))?;
push_with_hostname(&client, &hostname_str)
.map_err(|e| format!("Push phase failed: {}", e))?;
Ok(json!({
"success": true,
"message": "Successfully completed full sync"
})
.to_string())
})();
sync_result_to_jstring(env, "syncBoth", result)
})
}
/// Get the sync directory path
#[no_mangle]
pub extern "C" fn Java_net_activitywatch_android_SyncInterface_getSyncDir(
mut env: JNIEnv,
_class: JClass,
) -> jstring {
jni_guard(
&mut env,
"getSyncDir",
|env| match crate::dirs::get_sync_dir() {
Ok(path) => {
let path_str = path.to_string_lossy().to_string();
let response = json!({
"success": true,
"path": path_str
})
.to_string();
rust_string_to_jstring(env, response)
}
Err(e) => {
let msg = format!("Failed to get sync dir: {}", e);
error!("getSyncDir error: {}", msg);
rust_string_to_jstring(env, sync_error_json(&msg))
}
},
)
}

215
aw-sync/src/dirs.rs Normal file
View File

@@ -0,0 +1,215 @@
use dirs::home_dir;
use std::error::Error;
use std::fs;
#[cfg(any(target_os = "android", test))]
use std::path::Path;
use std::path::PathBuf;
/// Resolve the instance profile.
/// `--profile` wins, then `AW_PROFILE`, then `--testing` → `"testing"`, else `"default"`.
#[allow(dead_code)] // used by the aw-sync binary; the lib copy is unused
pub fn resolve_profile(
cli_profile: Option<&str>,
testing: bool,
env_profile: Option<&str>,
) -> String {
if let Some(p) = cli_profile {
if !p.is_empty() {
return p.to_string();
}
}
if let Some(p) = env_profile {
if !p.is_empty() {
return p.to_string();
}
}
if testing {
"testing".to_string()
} else {
"default".to_string()
}
}
/// aw-sync's own config dir: `{appname}/aw-sync`.
///
/// Uses the same profile appname as aw-server so a named profile (e.g.
/// `research`) does not share prod's sync config. `testing` follows the
/// same new-root-plus-legacy-fallback rule as aw-server.
// TODO: add proper config support
#[cfg(not(target_os = "android"))]
#[allow(dead_code)]
pub fn get_config_dir() -> Result<PathBuf, Box<dyn Error>> {
let dir = sync_config_dir(&aw_server::dirs::appname())?;
fs::create_dir_all(&dir)?;
Ok(dir)
}
/// Path construction only — does not create directories (so tests stay off-disk).
#[cfg(not(target_os = "android"))]
fn sync_config_dir(appname: &str) -> Result<PathBuf, Box<dyn Error>> {
Ok(dirs::config_dir()
.ok_or("Unable to read user config dir")?
.join(appname)
.join("aw-sync"))
}
/// Path to the embedded/local aw-server config. On Android this is
/// `filesDir/config.toml` (same file ConfigManager and the server use).
#[allow(dead_code)]
pub fn get_server_config_path(testing: bool) -> Result<PathBuf, ()> {
let profile = aw_server::config::get_profile();
// If set_profile hasn't run yet, honour the legacy testing bool so
// `--testing` still finds the testing config (legacy or new root).
let effective = if profile == "default" && testing {
"testing"
} else {
profile
};
let filename = aw_server::config::config_filename(effective);
// Desktop: honour profile isolation (`activitywatch-<profile>/aw-server-rust`).
// Android: stay on filesDir — sibling XDG roots do not exist there, and
// master already reads this path for the embedded server's api_key (#666).
#[cfg(not(target_os = "android"))]
{
let dir = dirs::config_dir()
.ok_or(())?
.join(aw_server::dirs::appname_for(effective))
.join("aw-server-rust");
Ok(dir.join(filename))
}
#[cfg(target_os = "android")]
{
let dir = aw_server::dirs::get_config_dir()?;
Ok(dir.join(filename))
}
}
pub fn get_sync_dir() -> Result<PathBuf, Box<dyn Error>> {
// if AW_SYNC_DIR is set, use that
if let Ok(dir) = std::env::var("AW_SYNC_DIR") {
return Ok(PathBuf::from(dir));
}
let home_dir = home_dir().ok_or("Unable to read home_dir")?;
Ok(home_dir.join("ActivityWatchSync"))
}
/// SyncInterface.kt sets `XDG_DATA_HOME=$filesDir/data` before `loadLibrary`.
/// `libaw_sync.so` is a separate cdylib from `libaw_server.so`, so
/// `RustInterface.setDataDir` does not update this library's `ANDROID_DATA_DIR`.
/// Parent of that env var is the app filesDir (release, `.debug`, work profile).
#[cfg(any(target_os = "android", test))]
pub(crate) fn files_dir_from_xdg_data_home(xdg_data_home: &Path) -> Option<PathBuf> {
if xdg_data_home.file_name() != Some(std::ffi::OsStr::new("data")) {
return None;
}
let parent = xdg_data_home.parent()?;
if parent.as_os_str().is_empty() || parent == Path::new("/") {
return None;
}
Some(parent.to_path_buf())
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::Path;
#[test]
fn resolve_profile_cli_wins_over_env_and_testing() {
assert_eq!(
resolve_profile(Some("research"), true, Some("testing")),
"research"
);
}
#[test]
fn resolve_profile_env_wins_over_testing() {
assert_eq!(resolve_profile(None, true, Some("research")), "research");
}
#[test]
fn resolve_profile_testing_alias_and_default() {
assert_eq!(resolve_profile(None, true, None), "testing");
assert_eq!(resolve_profile(None, false, None), "default");
assert_eq!(resolve_profile(Some(""), false, Some("")), "default");
}
#[cfg(not(target_os = "android"))]
#[test]
fn sync_config_dir_is_isolated_per_named_profile() {
use aw_server::dirs::appname_for_in;
use std::fs;
let root = std::env::temp_dir().join(format!(
"aw-sync-profile-tests-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let data = root.join("data");
let config = root.join("config");
let cache = root.join("cache");
fs::create_dir_all(&data).unwrap();
fs::create_dir_all(&config).unwrap();
fs::create_dir_all(&cache).unwrap();
let default_app = appname_for_in("default", &data, &config, &cache);
let testing_app = appname_for_in("testing", &data, &config, &cache);
let research_app = appname_for_in("research", &data, &config, &cache);
assert_eq!(default_app, "activitywatch");
// Fresh setup: testing is a sibling root, not the shared one.
assert_eq!(testing_app, "activitywatch-testing");
assert_eq!(research_app, "activitywatch-research");
assert_ne!(testing_app, default_app);
assert_ne!(research_app, default_app);
let _ = fs::remove_dir_all(root);
}
#[test]
fn server_config_filename_default_and_research_are_bare() {
// Isolated roots use config.toml. Suffixed config-testing.toml is
// only for the legacy shared-root layout (dirs.rs tests).
assert_eq!(aw_server::config::config_filename("default"), "config.toml");
assert_eq!(
aw_server::config::config_filename("research"),
"config.toml"
);
}
#[cfg(not(target_os = "android"))]
#[test]
fn server_config_path_default_is_config_toml() {
let production = get_server_config_path(false).unwrap();
assert!(
production.ends_with("config.toml"),
"default should read config.toml, got {production:?}"
);
}
#[test]
fn xdg_data_home_parent_is_files_dir() {
let debug = Path::new("/data/user/0/net.activitywatch.android.debug/files/data");
assert_eq!(
files_dir_from_xdg_data_home(debug).as_deref(),
Some(Path::new(
"/data/user/0/net.activitywatch.android.debug/files"
))
);
let work_profile = Path::new("/data/user/10/net.activitywatch.android/files/data");
assert_eq!(
files_dir_from_xdg_data_home(work_profile).as_deref(),
Some(Path::new("/data/user/10/net.activitywatch.android/files"))
);
}
#[test]
fn rejects_non_data_leaf_and_root() {
assert!(files_dir_from_xdg_data_home(Path::new("/data")).is_none());
assert!(files_dir_from_xdg_data_home(Path::new("/")).is_none());
assert!(files_dir_from_xdg_data_home(Path::new("data")).is_none());
assert!(files_dir_from_xdg_data_home(Path::new("/tmp/config")).is_none());
}
}

24
aw-sync/src/lib.rs Normal file
View File

@@ -0,0 +1,24 @@
#[macro_use]
extern crate log;
extern crate chrono;
extern crate serde;
extern crate serde_json;
mod sync;
pub use sync::create_datastore;
pub use sync::sync_datastores;
pub use sync::sync_run;
pub use sync::SyncSpec;
mod sync_wrapper;
pub use sync_wrapper::{pull, pull_all};
pub use sync_wrapper::{push, push_with_hostname};
mod accessmethod;
pub use accessmethod::AccessMethod;
mod dirs;
mod util;
#[cfg(target_os = "android")]
pub mod android;

351
aw-sync/src/main.rs Normal file
View File

@@ -0,0 +1,351 @@
#![cfg(feature = "cli")]
// What needs to be done:
// - [x] Setup local sync bucket
// - [x] Import local buckets and sync events from aw-server (either through API or through creating a read-only Datastore)
// - [x] Import buckets and sync events from remotes
// - [x] Add CLI arguments
// - [x] For which local server to use
// - [x] For which sync dir to use
// - [x] Date to start syncing from
#[macro_use]
extern crate log;
extern crate chrono;
extern crate serde;
extern crate serde_json;
use chrono::{DateTime, Utc};
use clap::{Parser, Subcommand};
use std::error::Error;
use std::path::PathBuf;
use std::sync::mpsc::{channel, RecvTimeoutError};
use std::time::Duration;
use aw_client_rust::blocking::AwClient;
mod accessmethod;
mod dirs;
mod sync;
mod sync_wrapper;
mod util;
#[derive(Parser)]
#[clap(version = "0.1", author = "Erik Bjäreholt")]
struct Opts {
#[clap(subcommand)]
command: Option<Commands>,
/// Host of instance to connect to.
#[clap(long, default_value = "127.0.0.1")]
host: String,
/// Port of instance to connect to.
#[clap(long)]
port: Option<u16>,
/// Path to the local aw-server config file.
#[clap(short = 'c', long = "config")]
config: Option<PathBuf>,
/// Named instance profile (e.g. "default", "testing", "research").
/// `--testing` is an alias for `--profile testing`. Also reads AW_PROFILE,
/// which aw-qt exports for the modules it spawns.
#[clap(long)]
profile: Option<String>,
/// Convenience option for using the default testing host and port.
/// Alias for `--profile testing`.
#[clap(long)]
testing: bool,
/// Full path to sync directory.
/// If not specified, use AW_SYNC_DIR env var, or default to ~/ActivityWatchSync
#[clap(long)]
sync_dir: Option<PathBuf>,
/// Enable debug logging.
#[clap(long)]
verbose: bool,
}
#[derive(Subcommand)]
enum Commands {
/// Daemon subcommand
/// Starts aw-sync as a daemon, which will sync every 5 minutes.
Daemon {
/// Date to start syncing from.
/// If not specified, start from beginning.
/// Format: YYYY-MM-DD
#[clap(long, value_parser=parse_start_date)]
start_date: Option<DateTime<Utc>>,
/// Specify buckets to sync using a comma-separated list.
/// By default, all buckets are synced.
#[clap(long, value_parser=parse_list)]
buckets: Option<Vec<String>>,
/// Mode to sync in. Can be "push", "pull", or "both".
/// Defaults to "both".
#[clap(long, default_value = "both")]
mode: sync::SyncMode,
/// Full path to sync db file
/// Useful for syncing buckets from a specific db file in the sync directory.
/// Must be a valid absolute path to a file in the sync directory.
#[clap(long)]
sync_db: Option<PathBuf>,
},
/// Sync subcommand
///
/// Syncs data between local aw-server and sync directory.
/// First pulls remote buckets from the sync directory to the local aw-server.
/// Then pushes local buckets from the aw-server to the local sync directory.
Sync {
/// Host(s) to pull from, comma separated. Will pull from all hosts if not specified.
#[clap(long, value_parser=parse_list)]
host: Option<Vec<String>>,
/// Date to start syncing from.
/// If not specified, start from beginning.
/// Format: YYYY-MM-DD
#[clap(long, value_parser=parse_start_date)]
start_date: Option<DateTime<Utc>>,
/// Specify buckets to sync using a comma-separated list.
/// By default, all buckets are synced.
#[clap(long, value_parser=parse_list)]
buckets: Option<Vec<String>>,
/// Mode to sync in. Can be "push", "pull", or "both".
/// Defaults to "both". Implies advanced sync mode (see below),
/// since the simple host-based sync mode always does both.
#[clap(long)]
mode: Option<sync::SyncMode>,
/// Full path to sync db file
/// Useful for syncing buckets from a specific db file in the sync directory.
/// Must be a valid absolute path to a file in the sync directory.
#[clap(long)]
sync_db: Option<PathBuf>,
},
/// List buckets and their sync status.
List {},
}
fn parse_start_date(arg: &str) -> Result<DateTime<Utc>, chrono::ParseError> {
chrono::NaiveDate::parse_from_str(arg, "%Y-%m-%d")
.map(|nd| nd.and_time(chrono::NaiveTime::MIN).and_utc())
}
fn parse_list(arg: &str) -> Result<Vec<String>, clap::Error> {
// If the argument is empty or just whitespace, return an empty Vec
// This handles the case when --buckets is used without a value
if arg.trim().is_empty() {
return Ok(vec![]);
}
// Otherwise, split by comma, trim each part, and filter out empty strings
Ok(arg
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect())
}
fn main() -> Result<(), Box<dyn Error>> {
let opts: Opts = Opts::parse();
let verbose = opts.verbose;
info!("Started aw-sync...");
let env_profile = std::env::var("AW_PROFILE").ok();
let profile = dirs::resolve_profile(
opts.profile.as_deref(),
opts.testing,
env_profile.as_deref(),
);
aw_server::dirs::validate_profile(&profile)
.unwrap_or_else(|e| panic!("Invalid profile name: {e}"));
aw_server::config::set_profile(profile.clone());
aw_server::logging::setup_logger("aw-sync", &profile, verbose)?;
if opts.testing && opts.profile.as_deref().is_some_and(|p| p != "testing") {
warn!("--testing and --profile {profile} both given; --profile wins, using '{profile}'");
}
if profile != "default" {
info!("Running aw-sync with profile '{profile}'");
}
// if sync_dir, set env var
if let Some(sync_dir) = opts.sync_dir {
if !sync_dir.is_absolute() {
Err("Sync dir must be absolute")?
}
info!("Using sync dir: {}", sync_dir.display());
std::env::set_var("AW_SYNC_DIR", sync_dir);
}
// Named profiles (and `--profile testing` without `--testing`) must use the
// profile's own server config / port default, not the CLI testing bool.
let testing = profile == "testing";
let server_config = if util::is_loopback_host(&opts.host) {
util::get_server_config(testing, opts.config.as_deref())?
} else {
util::ServerConfig::default_for(testing)
};
let port = opts.port.unwrap_or(server_config.port);
let api_key = server_config.api_key;
if api_key.is_some() {
info!("Using API key from server config for authentication");
}
let url_host = util::host_for_url(&opts.host);
let client = AwClient::new_with_api_key(&url_host, port, "aw-sync", api_key)?;
// if opts.command is None, then we're using the default subcommand (Daemon)
match opts.command.unwrap_or(Commands::Daemon {
start_date: None,
buckets: None,
mode: sync::SyncMode::Both,
sync_db: None,
}) {
// Start daemon
Commands::Daemon {
start_date,
buckets,
mode,
sync_db,
} => {
info!("Starting daemon...");
let effective_buckets = buckets;
daemon(&client, start_date, effective_buckets, sync_db, mode)?;
}
// Perform sync
Commands::Sync {
host,
start_date,
buckets,
mode,
sync_db,
} => {
let effective_buckets = buckets;
// If advanced options are provided, use advanced sync mode
if start_date.is_some()
|| effective_buckets.is_some()
|| sync_db.is_some()
|| mode.is_some()
{
let sync_dir = dirs::get_sync_dir()?;
if let Some(db_path) = &sync_db {
info!("Using sync db: {}", db_path.display());
if !db_path.is_absolute() {
Err("Sync db path must be absolute")?
}
if !db_path.starts_with(&sync_dir) {
Err("Sync db path must be in sync directory")?
}
}
let sync_spec = sync::SyncSpec {
path: sync_dir,
path_db: sync_db,
buckets: effective_buckets,
start: start_date,
};
sync::sync_run(&client, &sync_spec, mode.unwrap_or(sync::SyncMode::Both))?
} else {
// Simple host-based sync mode (backwards compatibility)
// Pull
match host {
Some(hosts) => {
for host in hosts.iter() {
info!("Pulling from host: {}", host);
sync_wrapper::pull(host, &client)?;
}
}
None => {
info!("Pulling from all hosts");
sync_wrapper::pull_all(&client)?;
}
}
// Push
info!("Pushing local data");
sync_wrapper::push(&client)?
}
}
// List all buckets
Commands::List {} => sync::list_buckets(&client)?,
}
// Needed to give the datastores some time to commit before program is shut down.
// 100ms isn't actually needed, seemed to work fine with as little as 10ms, but I'd rather give
// it some wiggleroom.
std::thread::sleep(std::time::Duration::from_millis(100));
Ok(())
}
fn daemon(
client: &AwClient,
start_date: Option<DateTime<Utc>>,
buckets: Option<Vec<String>>,
sync_db: Option<PathBuf>,
mode: sync::SyncMode,
) -> Result<(), Box<dyn Error>> {
let (tx, rx) = channel();
ctrlc::set_handler(move || {
let _ = tx.send(());
})?;
let sync_dir = dirs::get_sync_dir()?;
if let Some(db_path) = &sync_db {
info!("Using sync db: {}", db_path.display());
if !db_path.is_absolute() {
Err("Sync db path must be absolute")?
}
if !db_path.starts_with(&sync_dir) {
Err("Sync db path must be in sync directory")?
}
}
let sync_spec = sync::SyncSpec {
path: sync_dir,
buckets,
path_db: sync_db,
start: start_date,
};
loop {
if let Err(e) = sync::sync_run(client, &sync_spec, mode) {
error!("Error during sync cycle: {}", e);
return Err(e);
}
info!("Sync pass done, sleeping for 5 minutes");
match rx.recv_timeout(Duration::from_secs(300)) {
Ok(_) | Err(RecvTimeoutError::Disconnected) => {
info!("Termination signal received, shutting down.");
break;
}
Err(RecvTimeoutError::Timeout) => {
// Continue the loop if the timeout occurs
}
}
}
Ok(())
}

568
aw-sync/src/sync.rs Normal file
View File

@@ -0,0 +1,568 @@
/// Basic syncing for ActivityWatch
/// Based on: https://github.com/ActivityWatch/aw-server/pull/50
///
/// This does not handle any direct peer interaction/connections/networking, it works as a "bring your own folder synchronizer".
///
/// It manages a sync-folder by syncing the aw-server datastore with a copy/staging datastore in the folder (one for each host).
/// The sync folder is then synced with remotes using Syncthing/Dropbox/whatever.
extern crate chrono;
extern crate reqwest;
extern crate serde_json;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use aw_client_rust::blocking::AwClient;
use chrono::{DateTime, Duration, Utc};
use aw_datastore::{Datastore, DatastoreError};
use aw_models::{Bucket, Event};
#[cfg(feature = "cli")]
use clap::ValueEnum;
use crate::accessmethod::AccessMethod;
#[derive(PartialEq, Eq, Copy, Clone)]
#[cfg_attr(feature = "cli", derive(ValueEnum))]
pub enum SyncMode {
Push,
Pull,
Both,
}
#[derive(Debug)]
pub struct SyncSpec {
/// Path of sync folder
pub path: PathBuf,
/// Path of sync db
/// If None, will use all
pub path_db: Option<PathBuf>,
/// Bucket IDs to sync
pub buckets: Option<Vec<String>>,
/// Start of time range to sync
pub start: Option<DateTime<Utc>>,
}
impl Default for SyncSpec {
fn default() -> Self {
// TODO: Better default path
let path = Path::new("/tmp/aw-sync").to_path_buf();
SyncSpec {
path,
path_db: None,
buckets: None,
start: None,
}
}
}
/// Performs a single sync pass
pub fn sync_run(
client: &AwClient,
sync_spec: &SyncSpec,
mode: SyncMode,
) -> Result<(), Box<dyn Error>> {
let info = client.get_info()?;
// FIXME: Here it is assumed that the device_id for the local server is the one used by
// aw-server-rust, which is not necessarily true (aw-server-python has seperate device_id).
// Therefore, this may sometimes fail to pick up the correct local datastore.
let device_id = info.device_id.as_str();
// FIXME: Bad device_id assumption?
let ds_localremote = setup_local_remote(sync_spec.path.as_path(), device_id)?;
let remote_dbfiles = crate::util::find_remotes_nonlocal(
sync_spec.path.as_path(),
device_id,
sync_spec.path_db.as_ref(),
)?;
// Log if remotes found
// TODO: Only log remotes of interest
if !remote_dbfiles.is_empty() {
info!(
"Found {} remote db files: {:?}",
remote_dbfiles.len(),
remote_dbfiles
);
}
// TODO: Check for compatible remote db version before opening
let ds_remotes: Vec<Datastore> = remote_dbfiles
.iter()
.map(|p| p.as_path())
.map(create_datastore)
.collect::<Result<Vec<_>, _>>()?;
if !ds_remotes.is_empty() {
info!(
"Found {} remote datastores: {:?}",
ds_remotes.len(),
ds_remotes
);
}
// Pull
if mode == SyncMode::Pull || mode == SyncMode::Both {
info!("Pulling...");
for ds_from in &ds_remotes {
sync_datastores(ds_from, client, false, None, sync_spec)?;
}
}
// Push local server buckets to sync folder
if mode == SyncMode::Push || mode == SyncMode::Both {
info!("Pushing...");
sync_datastores(client, &ds_localremote, true, Some(device_id), sync_spec)?;
}
// Close open database connections
for ds_from in &ds_remotes {
ds_from.close();
}
ds_localremote.close();
// Dropping also works to close the database connections, weirdly enough.
// Probably because once the database is dropped, the thread will stop,
// and then the Connection will be dropped, which closes the connection.
std::mem::drop(ds_remotes);
std::mem::drop(ds_localremote);
// NOTE: Will fail if db connections not closed (as it will open them again)
//list_buckets(&client, sync_spec.path.as_path());
Ok(())
}
#[allow(dead_code)]
pub fn list_buckets(client: &AwClient) -> Result<(), Box<dyn Error>> {
let sync_directory = crate::dirs::get_sync_dir().map_err(|_| "Could not get sync dir")?;
let sync_directory = sync_directory.as_path();
let info = client.get_info()?;
// FIXME: Incorrect device_id assumption?
let device_id = info.device_id.as_str();
let ds_localremote = setup_local_remote(sync_directory, device_id)?;
let remote_dbfiles = crate::util::find_remotes_nonlocal(sync_directory, device_id, None)?;
info!("Found remotes: {:?}", remote_dbfiles);
// TODO: Check for compatible remote db version before opening
let ds_remotes: Vec<Datastore> = remote_dbfiles
.iter()
.map(|p| p.as_path())
.map(create_datastore)
.collect::<Result<Vec<_>, _>>()?;
log_buckets(client)?;
log_buckets(&ds_localremote)?;
for ds_from in &ds_remotes {
log_buckets(ds_from)?;
}
Ok(())
}
fn setup_local_remote(path: &Path, device_id: &str) -> Result<Datastore, Box<dyn Error>> {
// FIXME: Don't run twice if already exists
fs::create_dir_all(path)?;
let remotedir = path.join(device_id);
fs::create_dir_all(&remotedir)?;
let dbfile = remotedir.join("test.db");
// Print a message if dbfile doesn't already exist
if !dbfile.exists() {
info!("Creating new database file: {}", dbfile.display());
}
let ds_localremote = create_datastore(&dbfile)?;
Ok(ds_localremote)
}
/// Open (or create) the sqlite datastore at `path`.
///
/// `Datastore::new` takes a `String`, so a non-UTF-8 path cannot be passed
/// through faithfully. Report that as an error rather than unwrapping (a panic
/// here aborts the app on Android, aw-android#220) and rather than lossily
/// converting it, which would silently open a *different* file than the caller
/// asked for.
pub fn create_datastore(path: &Path) -> Result<Datastore, String> {
let pathstr = path
.to_str()
.ok_or_else(|| format!("Sync database path is not valid UTF-8: {}", path.display()))?;
Ok(Datastore::new(pathstr.to_string(), false))
}
/// Returns the sync-destination bucket for a given bucket, creates it if it doesn't exist.
///
/// Returns an error rather than panicking on a datastore failure or on bucket
/// metadata of an unexpected shape: this runs inside a JNI call on Android,
/// where an unwind out of the `extern "C"` frame aborts the app
/// (ActivityWatch/aw-android#220).
fn get_or_create_sync_bucket(
bucket_from: &Bucket,
ds_to: &dyn AccessMethod,
is_push: bool,
) -> Result<Bucket, String> {
// On pull/import: derive the origin from $aw.sync.origin metadata (preferred) or the
// hostname (legacy fallback for buckets that predate the metadata field). On push-staging
// the bucket keeps its original ID and we do NOT stamp $aw.sync.origin — staging copies
// should not look like synced-from-remote buckets.
let (new_id, sync_origin) = if is_push {
(bucket_from.id.clone(), None)
} else {
// `split` always yields at least one item, so this cannot be None.
let orig_bucketid = bucket_from
.id
.split("-synced-from-")
.next()
.unwrap_or(bucket_from.id.as_str());
let origin = match bucket_from.data.get("$aw.sync.origin") {
Some(value) => value.as_str().map(str::to_string).ok_or_else(|| {
format!(
"Bucket '{}' has a non-string $aw.sync.origin: {}",
bucket_from.id, value
)
})?,
None => bucket_from.hostname.clone(),
};
(
format!("{orig_bucketid}-synced-from-{origin}"),
Some(origin),
)
};
match ds_to.get_bucket(new_id.as_str()) {
Ok(bucket) => Ok(bucket),
Err(DatastoreError::NoSuchBucket(_)) => {
let mut bucket_new = bucket_from.clone();
bucket_new.id = new_id.clone();
// Only stamp $aw.sync.origin on pull/import. The derived origin already handles
// the legacy case: hostname is used when the source bucket has no metadata field.
if let Some(origin) = sync_origin {
bucket_new
.data
.insert("$aw.sync.origin".to_string(), serde_json::json!(origin));
} else {
// Push path: strip any stale $aw.sync.origin that bucket_from may carry
// (e.g. if it was previously imported by a pull). Staging copies must
// never look like synced-from-remote buckets.
bucket_new.data.remove("$aw.sync.origin");
}
ds_to
.create_bucket(&bucket_new)
.map_err(|e| format!("Failed to create bucket '{new_id}': {e:?}"))?;
ds_to
.get_bucket(new_id.as_str())
.map_err(|e| format!("Failed to read back bucket '{new_id}': {e:?}"))
}
Err(e) => Err(format!("Failed to get bucket '{new_id}': {e:?}")),
}
}
/// Number of events fetched per page in the chunked-fetch loop in `sync_one`.
/// Reduced in tests so multi-page paths can be exercised with a small event count.
#[cfg(not(test))]
const BATCH_SIZE: usize = 5000;
#[cfg(test)]
const BATCH_SIZE: usize = 5;
/// Whether a bucket holds data synced from another host, rather than data
/// collected on this host.
///
/// The `-synced-from-<origin>` ID suffix is the marker, matching how
/// `get_or_create_sync_bucket` builds and parses these IDs. Note that
/// `$aw.sync.origin` cannot be used here: it is written on push-staging as well
/// as on import (see the FIXME on `sync_datastores`), so it is set on a host's
/// own exported buckets too and would make every bucket look second-hand.
///
/// `-synced-from-` is a **reserved token** in aw-sync's ID grammar, not merely a
/// convention: `get_or_create_sync_bucket` splits on it to recover the original
/// ID, so a first-hand bucket whose own ID contained it would already have that
/// ID truncated on import, independently of this check. Treating it as a marker
/// therefore adds no new failure mode. Issue #649 tracks moving provenance to
/// bucket metadata, which removes the dependency on the ID string entirely.
fn is_synced_bucket(bucket: &Bucket) -> bool {
bucket.id.contains("-synced-from-")
}
/// Syncs all buckets from `ds_from` to `ds_to` with `-synced` appended to the ID of the destination bucket.
///
/// Buckets that were themselves synced from another host are skipped in both
/// directions, so data is only ever exchanged first-hand.
///
/// is_push: a bool indicating if we're pushing local buckets to the sync dir
/// (as opposed to pulling from remotes)
/// src_did: source device ID
///
/// Returns an error instead of panicking: this is the step the abort in
/// ActivityWatch/aw-android#220 happens in (the sync directory and its
/// datastore are already created by the time the process dies), and on Android
/// it runs inside a JNI `extern "C"` frame where an unwind is an abort rather
/// than a caught exception.
pub fn sync_datastores(
ds_from: &dyn AccessMethod,
ds_to: &dyn AccessMethod,
is_push: bool,
src_did: Option<&str>,
sync_spec: &SyncSpec,
) -> Result<(), String> {
// FIXME: "-synced" should only be appended when synced to the local database, not to the
// staging area for local buckets.
info!("Syncing {:?} to {:?}", ds_from, ds_to);
let mut buckets_from: Vec<Bucket> = ds_from
.get_buckets()
.map_err(|e| format!("Failed to list buckets in {ds_from:?}: {e}"))?
.iter_mut()
// Never sync a bucket that is itself a copy synced from another host.
// A host must only ever offer data it collected itself. Without this,
// HOSTA's buckets reach HOSTB, are re-exported by HOSTB's next push, and
// come back to HOSTA as `<bucket>_HOSTA-synced-from-HOSTA` — a duplicate
// of the local bucket, so /timeline renders every event twice.
// See https://github.com/orgs/ActivityWatch/discussions/1373
.filter(|tup| {
if is_synced_bucket(tup.1) {
debug!(" - Skipping already-synced bucket '{}'", tup.1.id);
false
} else {
true
}
})
// Only filter buckets if specific bucket IDs are provided
.filter(|tup| {
let bucket = &tup.1;
if let Some(buckets) = &sync_spec.buckets {
// If "*" is in the buckets list or no buckets specified, sync all buckets
if buckets.iter().any(|b_id| b_id == "*") || buckets.is_empty() {
true
} else {
buckets.iter().any(|b_id| b_id == &bucket.id)
}
} else {
// By default, sync all buckets
true
}
})
.map(|tup| {
// TODO: Refuse to sync buckets without hostname/device ID set, or if set to 'unknown'
if tup.1.hostname == "unknown" {
// Only the push path carries a source device ID to substitute.
// On pull there is none, and continuing would give the bucket a
// `-synced-from-unknown` destination ID shared by every remote
// with that bucket ID, mixing events from unrelated devices.
// Refuse the sync instead (the previous code unwrapped the None
// here, which on Android aborts the whole app).
let did = src_did.ok_or_else(|| {
format!(
"Bucket '{}' has an unknown hostname/device ID and there is no source \
device ID to substitute; refusing to sync it without provenance",
tup.1.id
)
})?;
warn!(" ! Bucket hostname/device ID was invalid, setting to device ID/hostname");
tup.1.hostname = did.to_string();
}
Ok(tup.1.clone())
})
.collect::<Result<Vec<Bucket>, String>>()?;
// Log warning for buckets requested but not found
if let Some(buckets) = &sync_spec.buckets {
for b_id in buckets {
if !buckets_from.iter().any(|b| b.id == *b_id) {
error!(" ! Bucket \"{}\" not found in source datastore", b_id);
}
}
}
// Sync buckets in order of most recently updated
buckets_from.sort_by_key(|b| b.metadata.end);
for bucket_from in buckets_from {
let bucket_to = get_or_create_sync_bucket(&bucket_from, ds_to, is_push)?;
sync_one(ds_from, ds_to, bucket_from, bucket_to, sync_spec)?;
}
Ok(())
}
/// Syncs a single bucket from one datastore to another
fn sync_one(
ds_from: &dyn AccessMethod,
ds_to: &dyn AccessMethod,
bucket_from: Bucket,
bucket_to: Bucket,
sync_spec: &SyncSpec,
) -> Result<(), String> {
let eventcount_to_old = ds_to.get_event_count(bucket_to.id.as_str())?;
info!(" ⟳ Syncing bucket '{}'", bucket_to.id);
// Sync events
// FIXME: This should use bucket_to.metadata.end, but it doesn't because it doesn't work
// for empty buckets (Should be None, is Some(unknown_time))
// let resume_sync_at = bucket_to.metadata.end;
let most_recent_events = ds_to.get_events(bucket_to.id.as_str(), None, None, Some(1))?;
// If the destination bucket already has events, resume from where it left off.
// Otherwise (first sync of this bucket), fall back to sync_spec.start, if specified.
let resume_sync_at = most_recent_events
.first()
.map(|e| e.timestamp + e.duration)
.or(sync_spec.start);
if let Some(resume_time) = resume_sync_at {
info!(" + Resuming at {:?}", resume_time);
} else {
info!(" + Starting from beginning");
}
// Fetch events in bounded chunks to avoid OOM on devices with limited RAM (e.g. Android).
// get_events returns events in descending order (newest first), so we paginate backwards
// using the `end` parameter. Each chunk is written to `ds_to` as soon as it is fetched,
// so peak memory is O(BATCH_SIZE), not O(total events in the bucket).
//
// Each chunk is reversed before writing so events are inserted oldest-first (matching the
// insertion order in the source DB). This preserves consistent ID assignment across source
// and destination, which the sync tests rely on.
//
// Heartbeat semantics at the resume boundary: we use heartbeat() for the globally-oldest
// new event ONLY in the single-page case (pages_written == 0 when is_last_fetch fires).
// In that case dest's "last event" is still the pre-sync resume-boundary row, so
// heartbeat() can correctly merge an adjacent new event into it.
//
// In the multi-page case (pages_written > 0), newer pages have already been inserted and
// dest's "last event" is no longer the resume boundary — heartbeat() would compare against
// the wrong row and skip the merge anyway. Inserting the oldest event directly is correct.
let mut fetch_end: Option<DateTime<Utc>> = None;
let mut events_sent = 0usize;
let mut pages_written = 0u32;
loop {
let raw = ds_from.get_events(
bucket_from.id.as_str(),
resume_sync_at,
fetch_end,
Some(BATCH_SIZE as u64),
)?;
if raw.is_empty() {
break;
}
// Fewer events than requested means there's nothing older left to fetch.
let is_last_fetch = raw.len() < BATCH_SIZE;
let mut chunk: Vec<Event> = raw
.into_iter()
.map(|mut e| {
// Unset ID on events, as they are not globally unique
e.id = None;
e
})
.collect();
if !is_last_fetch {
// chunk is in DESC order (newest first); chunk.last() = oldest in this (full) page.
// Naively setting the next `end` to `oldest.timestamp - 1ns` silently drops events
// if the page happens to end mid-run of same-timestamp events: anything else at
// that exact timestamp would fall outside the next page's range. Guard against that
// by dropping ALL trailing events at `boundary_ts` from this page and leaving them
// for the next fetch (whose `end = boundary_ts` is inclusive, so it re-fetches
// the whole tied run at once).
//
// Note: we must drop the boundary event itself, not just its duplicates. Keeping
// one copy in this chunk while also setting `fetch_end = Some(boundary_ts)` (inclusive)
// would cause that event to be fetched again next page, producing a duplicate row.
// `raw` was non-empty and `chunk` is a 1:1 map of it, so both ends exist.
let (Some(oldest), Some(newest)) = (chunk.last(), chunk.first()) else {
return Err(format!(
"Empty event page while syncing bucket '{}'",
bucket_from.id
));
};
let boundary_ts = oldest.timestamp;
if newest.timestamp != boundary_ts {
// Safe to pop all boundary_ts events: the `if` guard ensures at least one
// earlier event (with a different timestamp) remains in the chunk.
while chunk.last().is_some_and(|e| e.timestamp == boundary_ts) {
chunk.pop();
}
fetch_end = Some(boundary_ts);
} else {
// Pathological case: every event in this full page shares the exact same
// timestamp, so we can't tell where the tied run ends without an unbounded
// query. This can't occur with AW's event model in practice (activity records
// span seconds+) — accept the page as-is rather than looping forever.
fetch_end = Some(boundary_ts - Duration::nanoseconds(1));
}
// Reverse to ASC order (oldest first) before inserting.
chunk.reverse();
events_sent += chunk.len();
pages_written += 1;
for batch in chunk.chunks(BATCH_SIZE) {
print!("({}/…)\r", events_sent);
ds_to.insert_events(bucket_to.id.as_str(), batch.to_vec())?;
}
} else {
// Last (oldest) page: process oldest-first to preserve ID ordering.
chunk.reverse(); // chunk is now ASC (oldest first)
// Use heartbeat() for the oldest event only in the single-page case:
// dest's "last event" is still the pre-sync resume-boundary row, so heartbeat()
// can correctly merge an adjacent new event into it (delta=0.0 → exact adjacency).
// In multi-page syncs, newer pages are already in dest, so heartbeat() would
// compare against the wrong row — insert directly instead.
if !chunk.is_empty() && pages_written == 0 {
let oldest = chunk.remove(0);
ds_to.heartbeat(bucket_to.id.as_str(), oldest, 0.0)?;
events_sent += 1;
}
// Insert the remaining events from the last page in ASC order.
if !chunk.is_empty() {
events_sent += chunk.len();
for batch in chunk.chunks(BATCH_SIZE) {
print!("({}/…)\r", events_sent);
ds_to.insert_events(bucket_to.id.as_str(), batch.to_vec())?;
}
}
break;
}
}
let eventcount_to_new = ds_to.get_event_count(bucket_to.id.as_str())?;
let new_events_count = eventcount_to_new - eventcount_to_old;
if new_events_count < 0 {
return Err(format!(
"Event count of bucket '{}' shrank during sync ({eventcount_to_old} -> {eventcount_to_new})",
bucket_to.id
));
}
if new_events_count > 0 {
info!(" = Synced {} new events", new_events_count);
} else {
info!(" ✓ Already up to date!");
}
Ok(())
}
fn log_buckets(ds: &dyn AccessMethod) -> Result<(), String> {
// Logs all buckets and some metadata for a given datastore
let buckets = ds.get_buckets()?;
info!("Buckets in {:?}:", ds);
for bucket in buckets.values() {
info!(" - {}", bucket.id.as_str());
info!(
" eventcount: {:?}",
ds.get_event_count(bucket.id.as_str())?
);
}
Ok(())
}

View File

@@ -0,0 +1,77 @@
use std::error::Error;
use std::fs;
use crate::sync::{sync_run, SyncMode, SyncSpec};
use aw_client_rust::blocking::AwClient;
pub fn pull_all(client: &AwClient) -> Result<(), Box<dyn Error>> {
let hostnames = crate::util::get_remotes()?;
for host in hostnames {
pull(&host, client)?
}
Ok(())
}
pub fn pull(host: &str, client: &AwClient) -> Result<(), Box<dyn Error>> {
client.wait_for_start()?;
// Path to the sync folder
// Sync folder is structured ./{hostname}/{device_id}/test.db
let sync_root_dir = crate::dirs::get_sync_dir().map_err(|_| "Could not get sync dir")?;
let sync_dir = sync_root_dir.join(host);
let dbs = fs::read_dir(&sync_dir)?
.filter_map(Result::ok)
.filter(|entry| entry.path().is_dir())
.map(|entry| fs::read_dir(entry.path()))
.filter_map(Result::ok)
.flatten()
.filter_map(Result::ok)
.filter(|entry| {
entry.path().is_file()
&& entry.path().extension().and_then(|os_str| os_str.to_str()) == Some("db")
})
.collect::<Vec<_>>();
// if more than one db, warn and use the largest one
if dbs.len() > 1 {
warn!(
"More than one db found in sync folder for host, choosing largest db {:?}",
dbs
);
}
let db = dbs
.into_iter()
.max_by_key(|entry| entry.metadata().map(|m| m.len()).unwrap_or(0))
.ok_or_else(|| format!("No db found in sync folder {:?}", sync_dir))?;
let sync_spec = SyncSpec {
path: sync_dir.clone(),
path_db: Some(db.path().clone()),
buckets: None, // Sync all buckets by default
start: None,
};
sync_run(client, &sync_spec, SyncMode::Pull)?;
Ok(())
}
pub fn push(client: &AwClient) -> Result<(), Box<dyn Error>> {
push_with_hostname(client, &client.hostname)
}
pub fn push_with_hostname(client: &AwClient, hostname: &str) -> Result<(), Box<dyn Error>> {
let sync_dir = crate::dirs::get_sync_dir()
.map_err(|_| "Could not get sync dir")?
.join(hostname);
let sync_spec = SyncSpec {
path: sync_dir,
path_db: None,
buckets: None, // Sync all buckets by default
start: None,
};
sync_run(client, &sync_spec, SyncMode::Push)?;
Ok(())
}

262
aw-sync/src/util.rs Normal file
View File

@@ -0,0 +1,262 @@
use std::error::Error;
use std::ffi::OsStr;
use std::fs;
use std::fs::File;
use std::io::Read;
use std::net::IpAddr;
use std::path::{Path, PathBuf};
// Only used by the binary (main.rs) and the Android entrypoint, so these are
// dead code in the plain library build.
#[allow(dead_code)]
pub struct ServerConfig {
pub port: u16,
pub api_key: Option<String>,
}
#[allow(dead_code)]
impl ServerConfig {
pub fn default_for(testing: bool) -> Self {
Self {
port: if testing { 5666 } else { 5600 },
api_key: None,
}
}
}
/// Returns the settings aw-sync needs from the selected aw-server config.
///
/// Also used on Android: the embedded server writes `config.toml` under
/// `filesDir`, and `get_client()` in `android.rs` must send the same
/// `[auth].api_key` or `/api/0/buckets` returns 401 (aw-android#247).
#[allow(dead_code)]
pub fn get_server_config(
testing: bool,
config_override: Option<&Path>,
) -> Result<ServerConfig, Box<dyn Error>> {
let path = match config_override {
Some(path) => path.to_path_buf(),
None => crate::dirs::get_server_config_path(testing)
.map_err(|_| "Could not get aw-server config path")?,
};
let default = ServerConfig::default_for(testing);
if !path.exists() {
return Ok(default);
}
let mut contents = String::new();
File::open(path)?.read_to_string(&mut contents)?;
let value: toml::Value = toml::from_str(&contents)?;
let port = value
.get("port")
.and_then(|v| v.as_integer())
.and_then(|v| u16::try_from(v).ok())
.unwrap_or(default.port);
let api_key = value
.get("auth")
.and_then(|a| a.get("api_key"))
.and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
.map(String::from);
Ok(ServerConfig { port, api_key })
}
/// Local config must never be read for a caller-selected remote target.
#[cfg(not(target_os = "android"))]
#[allow(dead_code)]
pub fn is_loopback_host(host: &str) -> bool {
host.eq_ignore_ascii_case("localhost")
|| host
.parse::<IpAddr>()
.is_ok_and(|address| address.is_loopback())
}
/// Add URL brackets around bare IPv6 literals.
#[cfg(not(target_os = "android"))]
#[allow(dead_code)]
pub fn host_for_url(host: &str) -> String {
match host.parse::<IpAddr>() {
Ok(IpAddr::V6(_)) => format!("[{host}]"),
_ => host.to_string(),
}
}
#[cfg(all(test, not(target_os = "android")))]
mod tests {
use super::{get_server_config, host_for_url, is_loopback_host};
use std::fs;
use std::time::{SystemTime, UNIX_EPOCH};
#[test]
fn reads_port_and_api_key_from_config_override() {
let config_path = std::env::temp_dir().join(format!(
"aw-sync-config-{}-{}.toml",
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
fs::write(
&config_path,
"port = 5611\n[auth]\napi_key = \"custom-key\"\n",
)
.unwrap();
let config = get_server_config(false, Some(&config_path)).unwrap();
fs::remove_file(config_path).unwrap();
assert_eq!(config.port, 5611);
assert_eq!(config.api_key.as_deref(), Some("custom-key"));
}
#[test]
fn missing_config_override_uses_defaults() {
let config_path = std::env::temp_dir().join(format!(
"missing-aw-sync-config-{}.toml",
std::process::id()
));
let _ = fs::remove_file(&config_path);
let production = get_server_config(false, Some(&config_path)).unwrap();
let testing = get_server_config(true, Some(&config_path)).unwrap();
assert_eq!(production.port, 5600);
assert!(production.api_key.is_none());
assert_eq!(testing.port, 5666);
assert!(testing.api_key.is_none());
}
#[test]
fn commented_or_empty_api_key_is_absent() {
let config_path = std::env::temp_dir().join(format!(
"aw-sync-config-commented-{}-{}.toml",
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
fs::write(
&config_path,
"port = 5600\n[auth]\n#api_key = \"secret\"\napi_key = \"\"\n",
)
.unwrap();
let config = get_server_config(false, Some(&config_path)).unwrap();
fs::remove_file(config_path).unwrap();
assert!(config.api_key.is_none());
}
#[test]
fn recognizes_only_loopback_hosts() {
for host in ["127.0.0.1", "127.0.0.2", "::1", "localhost", "LOCALHOST"] {
assert!(is_loopback_host(host));
}
for host in ["example.com", "192.0.2.1", "localhost.example.com"] {
assert!(!is_loopback_host(host));
}
}
#[test]
fn brackets_bare_ipv6_hosts_for_urls() {
assert_eq!(host_for_url("::1"), "[::1]");
assert_eq!(host_for_url("2001:db8::1"), "[2001:db8::1]");
assert_eq!(host_for_url("127.0.0.1"), "127.0.0.1");
assert_eq!(host_for_url("localhost"), "localhost");
}
}
/// Check if a directory contains a .db file
fn contains_db_file(dir: &std::path::Path) -> bool {
fs::read_dir(dir)
.ok()
.map(|entries| {
entries.filter_map(Result::ok).any(|entry| {
entry
.path()
.extension()
.map(|ext| ext == "db")
.unwrap_or(false)
})
})
.unwrap_or(false)
}
/// Check if a directory contains a subdirectory that contains a .db file
fn contains_subdir_with_db_file(dir: &std::path::Path) -> bool {
fs::read_dir(dir)
.ok()
.map(|entries| {
entries
.filter_map(Result::ok)
.any(|entry| entry.path().is_dir() && contains_db_file(&entry.path()))
})
.unwrap_or(false)
}
/// Return all remotes in the sync folder
/// Only returns folders that match ./{host}/{device_id}/*.db
// TODO: share logic with find_remotes and find_remotes_nonlocal
pub fn get_remotes() -> Result<Vec<String>, Box<dyn Error>> {
let sync_root_dir = crate::dirs::get_sync_dir()?;
fs::create_dir_all(&sync_root_dir)?;
let hostnames = fs::read_dir(sync_root_dir)?
.filter_map(Result::ok)
.filter(|entry| entry.path().is_dir() && contains_subdir_with_db_file(&entry.path()))
.filter_map(|entry| {
entry
.path()
.file_name()
.and_then(|os_str| os_str.to_str().map(String::from))
})
.collect();
info!("Found remotes: {:?}", hostnames);
Ok(hostnames)
}
/// Returns a list of all remote dbs
///
/// I/O errors are propagated rather than unwrapped (a panic here aborts the app
/// on Android, ActivityWatch/aw-android#220) and rather than skipped: silently
/// dropping a host directory we failed to read would report a successful sync
/// that quietly omitted that host's data.
fn find_remotes(sync_directory: &Path) -> std::io::Result<Vec<PathBuf>> {
let mut dbs = Vec::new();
for entry in fs::read_dir(sync_directory)? {
let hostdir = entry?.path();
if !hostdir.is_dir() {
continue;
}
for entry in fs::read_dir(&hostdir)? {
let path = entry?.path();
if path.extension().unwrap_or_else(|| OsStr::new("")) == "db" {
dbs.push(path);
}
}
}
Ok(dbs)
}
/// Returns a list of all remotes, excluding local ones
pub fn find_remotes_nonlocal(
sync_directory: &Path,
device_id: &str,
sync_db: Option<&PathBuf>,
) -> std::io::Result<Vec<PathBuf>> {
let remotes_all = find_remotes(sync_directory)?;
Ok(remotes_all
.into_iter()
// Filter out own remote
.filter(|path| !path.to_string_lossy().contains(device_id))
// If sync_db is Some, return only remotes in that path
.filter(|path| {
if let Some(sync_db) = sync_db {
path.starts_with(sync_db)
} else {
true
}
})
.collect())
}

13
aw-sync/test-server.sh Executable file
View File

@@ -0,0 +1,13 @@
#!/bin/bash
# port and db used for testing instance
# If port already set, use that, otherwise, use 5667
PORT=${PORT:-5667}
DBPATH=/tmp/aw-server-rust-sync-testing/
mkdir -p $DBPATH
# Set up an isolated ActivityWatch instance
pushd ..
cargo run --bin aw-server -- --testing --port $PORT --dbpath $DBPATH/data.db --no-legacy-import --verbose

78
aw-sync/test-sync-pull.sh Executable file
View File

@@ -0,0 +1,78 @@
#!/bin/bash
# exit on fail
set -e
# get script path
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
pushd $SCRIPTPATH
# port used for testing instance
PORT=5667
# if server already running on port 5667, don't start again
if [ "$(lsof -i:$PORT -sTCP:LISTEN -t)" ]; then
echo "ActivityWatch server already running on port $PORT, using that."
else
# Set up an isolated ActivityWatch instance
./test-server.sh $PORT &
SERVER_PID=$!
fi
sleep 1;
SYNCROOTDIR="$HOME/ActivityWatchSync"
function sync_host() {
host=$1
SYNCDIR="$SYNCROOTDIR/$host"
dbs=$(find $SYNCDIR -name "*.db")
for db in $dbs; do
# workaround to avoid trying to sync empty database files (size 45056)
if [ "$(stat -c%s $db)" -lt 50000 ]; then
continue
fi
AWSYNC_ARGS="--port $PORT"
AWSYNC_ARGS_ADV="--sync-dir $SYNCDIR --sync-db $db"
BUCKETS="aw-watcher-window_$host,aw-watcher-afk_$host"
echo "Syncing $db to $host"
cargo run --bin aw-sync -- $AWSYNC_ARGS sync-advanced $AWSYNC_ARGS_ADV --mode pull --buckets $BUCKETS
# TODO: If there are no buckets from the expected host, emit a warning at the end.
# (push-script should not have created them to begin with)
done
}
host=$1
# if no host given, sync all, otherwise sync only the given host
if [ -z "$host" ]; then
echo "Syncing all hosts"
sleep 0.5
# For each host in the sync directory, pull the data from each database file using aw-sync
# Use `find` to get all directories in the sync directory
hostnames=$(find $SYNCROOTDIR -maxdepth 1 -type d -exec basename {} \;)
# filter out "erb-m2.local"
hostnames=$(echo $hostnames | tr ' ' '\n' | grep -v "erb-m2.local")
# filter out folder not containing subfolders with .db files
for host in $hostnames; do
if [ ! "$(find $SYNCROOTDIR/$host -name "*.db")" ]; then
hostnames=$(echo $hostnames | tr ' ' '\n' | grep -v $host)
fi
done
# Sync each host, file-by-file
for host in $hostnames; do
sync_host $host
done
else
echo "Syncing host $1"
sleep 0.5
sync_host $host
fi
# kill aw-server-rust (if started by us)
if [ "$SERVER_PID" ]; then
kill $SERVER_PID
fi

32
aw-sync/test-sync-push.sh Executable file
View File

@@ -0,0 +1,32 @@
#!/bin/bash
# Helper script meant to be used to test aw-sync
# Example of a single-entry for cronjobs and the like
# exit on fail
set -e
# on Linux, use `hostnamectl`, on macOS, use `hostname`
if [ -x "$(command -v hostnamectl)" ]; then
HOSTNAME=$(hostnamectl --static)
else
HOSTNAME=$(hostname)
fi
# TODO: Fetch in a cross-platform way (from aw-client command output?)
AWSERVERCONF=~/.config/activitywatch/aw-server/aw-server.toml
# trim everything in file AWSERVERCONF before '[server-testing]' section
# grep for the aw-server port in aw-server.toml
# if config doesn't exist, assume 5600
if [ -f "$AWSERVERCONF" ]; then
PORT=$(sed '/\[server-testing\]/,/\[.*\]/{//!d}' $AWSERVERCONF | grep -oP 'port = "\K[0-9]+')
else
PORT=5600
fi
SYNCDIR="$HOME/ActivityWatchSync/$HOSTNAME"
AWSYNC_ARGS="--port $PORT"
AWSYNC_ARGS_ADV="--sync-dir $SYNCDIR"
# NOTE: Only sync window and AFK buckets, for now
cargo run --bin aw-sync --release -- $AWSYNC_ARGS sync-advanced $AWSYNC_ARGS_ADV --mode push --buckets aw-watcher-window_$HOSTNAME,aw-watcher-afk_$HOSTNAME

544
aw-sync/tests/sync.rs Normal file
View File

@@ -0,0 +1,544 @@
#[macro_use]
extern crate log;
extern crate aw_sync;
#[cfg(test)]
mod sync_tests {
use std::collections::HashMap;
use std::path::Path;
use chrono::{DateTime, Duration, Utc};
use aw_datastore::{Datastore, DatastoreError};
use aw_models::{Bucket, Event};
use aw_sync::{create_datastore, AccessMethod, SyncSpec};
struct TestState {
ds_src: Datastore,
ds_dest: Datastore,
}
fn init_teststate() -> TestState {
TestState {
ds_src: Datastore::new_in_memory(false),
ds_dest: Datastore::new_in_memory(false),
}
}
fn create_bucket(ds: &Datastore, n: i32) -> String {
// Create a bucket
let bucket_id = format!("bucket-{n}");
let bucket_jsonstr = format!(
r#"{{
"id": "{bucket_id}",
"type": "test",
"hostname": "device-{n}",
"client": "test"
}}"#
);
let bucket: Bucket = serde_json::from_str(&bucket_jsonstr).unwrap();
match ds.create_bucket(&bucket) {
Ok(()) => (),
Err(e) => match e {
DatastoreError::BucketAlreadyExists(_) => {
debug!("bucket already exists, skipping");
}
e => panic!("woops! {e:?}"),
},
};
bucket_id
}
fn create_event(data_str: &str) -> Event {
// A workaround needed because otherwise events might get same timestamp if
// call is repeated quickly on platforms with a low-precision clock.
std::thread::sleep(std::time::Duration::from_millis(5));
let timestamp: DateTime<Utc> = Utc::now();
let event_jsonstr = format!(
r#"{{
"timestamp": "{}",
"duration": 0,
"data": {{"test": {} }}
}}"#,
timestamp.to_rfc3339(),
data_str
);
serde_json::from_str(&event_jsonstr).unwrap()
}
fn create_events(ds: &Datastore, bucket_id: &str, n: i64) {
let events: Vec<Event> = (0..n)
.map(|i| create_event(format!("{i}").as_str()))
.collect::<Vec<Event>>();
ds.insert_events(bucket_id, &events[..]).unwrap();
ds.force_commit().unwrap();
}
fn get_all_buckets(datastores: Vec<&Datastore>) -> Vec<(&Datastore, Bucket)> {
let mut all_buckets: Vec<(&Datastore, Bucket)> = Vec::new();
for ds in datastores {
let buckets = ds.get_buckets().unwrap();
for bucket in buckets.values() {
all_buckets.push((ds, bucket.clone()));
}
}
all_buckets
}
fn get_all_buckets_map(datastores: Vec<&Datastore>) -> HashMap<String, (&Datastore, Bucket)> {
let all_buckets = get_all_buckets(datastores);
all_buckets
.iter()
.cloned()
.map(|(ds, b)| (b.id.clone(), (ds, b)))
.collect()
}
/// A datastore failure must be *returned*, never panicked.
///
/// On Android the sync step runs inside a JNI `extern "C"` frame, where an
/// unwinding panic aborts the whole app process rather than surfacing as an
/// exception — the SIGABRT in ActivityWatch/aw-android#220. `sync_datastores`
/// used to `unwrap()` every datastore call, so any failure here was fatal.
#[test]
fn test_unusable_datastore_returns_error_instead_of_panicking() {
let state = init_teststate();
create_bucket(&state.ds_src, 0);
// A datastore whose database file cannot be opened: its worker is gone,
// so every request fails.
let ds_broken = create_datastore(Path::new(
"/nonexistent-directory-for-aw-sync-tests/test.db",
))
.expect("path is valid UTF-8");
let result = aw_sync::sync_datastores(
&state.ds_src,
&ds_broken,
true,
Some("device-0"),
&SyncSpec::default(),
);
assert!(
result.is_err(),
"an unusable destination datastore must return Err, got {result:?}"
);
}
/// Bucket metadata of an unexpected shape must also be an error, not a panic:
/// `$aw.sync.origin` is read from data written by another host, so it is not
/// under this host's control.
#[test]
fn test_non_string_sync_origin_returns_error_instead_of_panicking() {
let state = init_teststate();
let bucket: Bucket = serde_json::from_str(
r#"{
"id": "bucket-weird",
"type": "test",
"hostname": "device-0",
"client": "test",
"data": {"$aw.sync.origin": 42}
}"#,
)
.unwrap();
state.ds_src.create_bucket(&bucket).unwrap();
let result = aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false, // pull: this is the path that reads $aw.sync.origin
None,
&SyncSpec::default(),
);
let err = result.expect_err("a non-string $aw.sync.origin must return Err");
assert!(
err.contains("$aw.sync.origin"),
"error should name the offending field, got: {err}"
);
}
/// A pulled bucket whose hostname is the "unknown" sentinel has no provenance,
/// and pull passes no source device ID to substitute. Continuing would map
/// every such bucket from every remote onto one `-synced-from-unknown`
/// destination, mixing events from unrelated devices — so refuse the sync.
/// (The old code unwrapped the `None` here, i.e. aborted the app on Android.)
#[test]
fn test_unknown_hostname_on_pull_returns_error() {
let state = init_teststate();
let bucket: Bucket = serde_json::from_str(
r#"{
"id": "bucket-unknown-host",
"type": "test",
"hostname": "unknown",
"client": "test"
}"#,
)
.unwrap();
state.ds_src.create_bucket(&bucket).unwrap();
let result = aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false, // pull: no source device ID is passed
None,
&SyncSpec::default(),
);
let err = result.expect_err("an unknown hostname on pull must return Err");
assert!(
err.contains("bucket-unknown-host"),
"error should name the bucket, got: {err}"
);
// On push the source device ID is known, so the same bucket syncs fine.
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
true,
Some("device-0"),
&SyncSpec::default(),
)
.unwrap();
}
#[test]
fn test_buckets_created() {
// TODO: Split up this test
let state = init_teststate();
create_bucket(&state.ds_src, 0);
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let buckets_src: HashMap<String, Bucket> = state.ds_src.get_buckets().unwrap();
let buckets_dest: HashMap<String, Bucket> = state.ds_dest.get_buckets().unwrap();
assert!(buckets_src.len() == buckets_dest.len());
}
/// On pull (is_push=false), the destination bucket should have $aw.sync.origin set to
/// the source bucket's hostname. On push (is_push=true), $aw.sync.origin must NOT be
/// written to the staging copy.
#[test]
fn test_sync_origin_metadata() {
let state = init_teststate();
let bucket_id = create_bucket(&state.ds_src, 0);
// --- push: staging copy must have no $aw.sync.origin ---
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
true, // is_push
Some("device-0"),
&SyncSpec::default(),
)
.unwrap();
let push_buckets = state.ds_dest.get_buckets().unwrap();
let push_bucket = push_buckets.get(&bucket_id).expect("push bucket not found");
assert!(
!push_bucket.data.contains_key("$aw.sync.origin"),
"push-staging bucket must not have $aw.sync.origin, got: {:?}",
push_bucket.data
);
// --- pull: imported bucket must have $aw.sync.origin set to the hostname ---
let ds_pull_dest = Datastore::new_in_memory(false);
aw_sync::sync_datastores(
&state.ds_src,
&ds_pull_dest,
false, // is_push (pull)
None,
&SyncSpec::default(),
)
.unwrap();
let pull_buckets = ds_pull_dest.get_buckets().unwrap();
let pull_bucket_id = format!("{bucket_id}-synced-from-device-0");
let pull_bucket = pull_buckets
.get(&pull_bucket_id)
.expect("pull bucket not found");
let origin = pull_bucket
.data
.get("$aw.sync.origin")
.expect("pull bucket must have $aw.sync.origin")
.as_str()
.expect("$aw.sync.origin must be a string");
assert_eq!(
origin, "device-0",
"$aw.sync.origin should match source hostname"
);
}
/// When bucket_from was previously pulled (carries $aw.sync.origin in its data), pushing it
/// to staging must strip the stale field so downstream devices don't trust a wrong origin.
#[test]
fn test_push_strips_stale_sync_origin() {
let state = init_teststate();
// Create a source bucket that already has $aw.sync.origin (simulating a
// previously-pulled bucket being pushed back to staging).
let bucket_id = "bucket-with-origin".to_string();
let bucket: Bucket = serde_json::from_value(serde_json::json!({
"id": bucket_id,
"type": "test",
"hostname": "device-0",
"client": "test",
"data": { "$aw.sync.origin": "some-other-host" }
}))
.unwrap();
state.ds_src.create_bucket(&bucket).unwrap();
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
true, // is_push
Some("device-0"),
&SyncSpec::default(),
)
.unwrap();
let push_buckets = state.ds_dest.get_buckets().unwrap();
let push_bucket = push_buckets.get(&bucket_id).expect("push bucket not found");
assert!(
!push_bucket.data.contains_key("$aw.sync.origin"),
"push-staging bucket must not carry stale $aw.sync.origin from a previously-pulled source, got: {:?}",
push_bucket.data
);
}
fn check_synced_buckets_equal_to_src(all_buckets_map: &HashMap<String, (&Datastore, Bucket)>) {
for (ds, bucket) in all_buckets_map.values() {
if bucket.id.contains("-synced") {
let bucket_src_id = bucket.id.split("-synced-").next().unwrap();
let (ds_src, bucket_src) = all_buckets_map.get(bucket_src_id).unwrap();
let events_synced = ds.get_events(bucket.id.as_str(), None, None, None).unwrap();
let events_src = ds_src
.get_events(bucket_src.id.as_str(), None, None, None)
.unwrap();
println!("{events_synced:?}");
println!("{events_src:?}");
assert!(events_synced == events_src);
}
}
}
#[test]
fn test_one_updated_event() {
// This tests the syncing of one single event that is then updated by a heartbeat after the
// first sync pass.
let state = init_teststate();
let bucket_id = create_bucket(&state.ds_src, 0);
state
.ds_src
.heartbeat(bucket_id.as_str(), create_event("1"), 1.0)
.unwrap();
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
let all_buckets_map = get_all_buckets_map(all_datastores);
// Check that all synced buckets are identical to source bucket
check_synced_buckets_equal_to_src(&all_buckets_map);
// Add some more events
state
.ds_src
.heartbeat(bucket_id.as_str(), create_event("1"), 1.0)
.unwrap();
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
// Check again that new events were indeed synced
check_synced_buckets_equal_to_src(&all_buckets_map);
}
#[test]
fn test_events() {
let state = init_teststate();
let bucket_id = create_bucket(&state.ds_src, 0);
create_events(&state.ds_src, bucket_id.as_str(), 10);
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
let all_buckets_map = get_all_buckets_map(all_datastores);
// Check that all synced buckets are identical to source bucket
check_synced_buckets_equal_to_src(&all_buckets_map);
// Add some more events
create_events(&state.ds_src, bucket_id.as_str(), 10);
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
// Check again that new events were indeed synced
check_synced_buckets_equal_to_src(&all_buckets_map);
}
#[test]
fn test_sync_resume_after_partial_sync() {
// Verify that resuming an interrupted sync picks up where it left off.
// This exercises the resume_sync_at path in the chunked-fetch loop.
let state = init_teststate();
let bucket_id = create_bucket(&state.ds_src, 0);
create_events(&state.ds_src, bucket_id.as_str(), 15);
// First sync pass
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores_1: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
check_synced_buckets_equal_to_src(&get_all_buckets_map(all_datastores_1));
// Add more events and sync again — verifies resume logic
create_events(&state.ds_src, bucket_id.as_str(), 15);
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores_2: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
check_synced_buckets_equal_to_src(&get_all_buckets_map(all_datastores_2));
}
#[test]
fn test_sync_multipage_no_duplicates() {
// Verify that multi-page syncs (more events than BATCH_SIZE) produce no duplicates.
// In tests, BATCH_SIZE=5, so 12 events triggers 3 pages (5+5+2).
// This exercises the boundary-event cursor logic that previously left one copy of
// the page-boundary event in the current chunk AND re-fetched it on the next page.
let state = init_teststate();
let bucket_id = create_bucket(&state.ds_src, 0);
create_events(&state.ds_src, bucket_id.as_str(), 12);
// Initial sync
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores_1: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
check_synced_buckets_equal_to_src(&get_all_buckets_map(all_datastores_1));
// Resume sync: add another 12 events (3 more pages) and sync again.
// Exercises multi-page resume path where heartbeat order matters.
create_events(&state.ds_src, bucket_id.as_str(), 12);
aw_sync::sync_datastores(
&state.ds_src,
&state.ds_dest,
false,
None,
&SyncSpec::default(),
)
.unwrap();
let all_datastores_2: Vec<&Datastore> = [&state.ds_src, &state.ds_dest].to_vec();
check_synced_buckets_equal_to_src(&get_all_buckets_map(all_datastores_2));
}
// TODO: Find a way to reuse this (previously used in an integration test)
fn setup_test(sync_directory: &Path) -> std::io::Result<Vec<Datastore>> {
let mut datastores: Vec<Datastore> = Vec::new();
for n in 0..2 {
let dspath = sync_directory.join(format!("test-remote-{n}.db"));
let ds_ = create_datastore(&dspath).expect("test db path is valid UTF-8");
let ds = &ds_ as &dyn AccessMethod;
// Create a bucket
// NOTE: Created with duplicate name to make sure it still works under such conditions
let bucket_jsonstr = format!(
r#"{{
"id": "bucket",
"type": "test",
"hostname": "device-{n}",
"client": "test"
}}"#
);
let bucket: Bucket = serde_json::from_str(&bucket_jsonstr)?;
match ds.create_bucket(&bucket) {
Ok(()) => (),
Err(e) => match e {
DatastoreError::BucketAlreadyExists(_) => {
debug!("bucket already exists, skipping");
}
e => panic!("woops! {e:?}"),
},
};
// Insert some testing events into the bucket
let events: Vec<Event> = (0..3)
.map(|i| {
let timestamp: DateTime<Utc> = Utc::now() + Duration::milliseconds(i * 10);
let event_jsonstr = format!(
r#"{{
"timestamp": "{}",
"duration": 0,
"data": {{"test": {} }}
}}"#,
timestamp.to_rfc3339(),
i
);
serde_json::from_str(&event_jsonstr).unwrap()
})
.collect::<Vec<Event>>();
ds.insert_events(bucket.id.as_str(), events).unwrap();
//let new_eventcount = ds.get_event_count(bucket.id.as_str(), None, None).unwrap();
//info!("Eventcount: {:?} ({} new)", new_eventcount, events.len());
datastores.push(ds_);
}
Ok(datastores)
}
}

View File

@@ -0,0 +1,164 @@
/// Regression test for the "synced-from-<own hostname>" duplication reported in
/// https://github.com/orgs/ActivityWatch/discussions/1373
///
/// aw-sync already refuses to import its *own* export (`find_remotes_nonlocal`
/// filters the local device_id out of the remote list). That guard only covers
/// the direct path. It does not cover the case where a host's data is laundered
/// through a *peer*: the peer imports HOSTA's buckets, then re-exports them as
/// part of its own push, and HOSTA imports them back as
/// `<bucket>_HOSTA-synced-from-HOSTA` sitting next to the real local bucket.
///
/// Both copies then render in /timeline, so every event is shown twice.
use std::path::PathBuf;
use std::sync::atomic::{AtomicU64, Ordering};
use aw_datastore::Datastore;
use aw_models::{Bucket, BucketMetadata};
use aw_sync::{sync_datastores, AccessMethod, SyncSpec};
// Tests in this binary run in parallel. Use a monotonic counter to guarantee
// each datastore gets a unique path even when two tests start within the same
// clock tick (Windows/macOS SystemTime resolution can be coarser than 1 ns,
// causing path collisions: "duplicate column name" on macOS, "database is
// locked" on Windows).
static DB_COUNTER: AtomicU64 = AtomicU64::new(0);
fn tmp_db(name: &str) -> PathBuf {
let mut p = std::env::temp_dir();
// Counter: within-run uniqueness. Timestamp: cross-run uniqueness when
// the PID is reused and the counter resets to 0. Same as #655.
let ts = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
p.push(format!(
"aw-sync-roundtrip-{}-{}-{}-{}.db",
std::process::id(),
name,
DB_COUNTER.fetch_add(1, Ordering::Relaxed),
ts,
));
p
}
fn datastore(name: &str) -> Datastore {
Datastore::new(tmp_db(name).to_str().unwrap().to_string(), false)
}
fn bucket(id: &str, hostname: &str) -> Bucket {
Bucket {
bid: None,
id: id.to_string(),
_type: "currentwindow".to_string(),
client: "aw-watcher-window".to_string(),
hostname: hostname.to_string(),
created: None,
data: serde_json::Map::new(),
metadata: BucketMetadata::default(),
events: None,
last_updated: None,
}
}
fn bucket_ids(ds: &dyn AccessMethod) -> Vec<String> {
let mut ids: Vec<String> = ds.get_buckets().unwrap().keys().cloned().collect();
ids.sort();
ids
}
/// Two hosts sharing a sync folder. Returns (a_local, b_export) after a full
/// push/pull round: HOSTA pushes, HOSTB pulls, HOSTB pushes, HOSTA pulls.
fn round_trip() -> (Datastore, Datastore) {
let spec = SyncSpec::default();
// HOSTA: local server datastore + the db it pushes into the sync folder.
let a_local = datastore("a-local");
let a_export = datastore("a-export");
// HOSTB: same.
let b_local = datastore("b-local");
let b_export = datastore("b-export");
// HOSTA collects some data locally.
a_local
.create_bucket(&bucket("aw-watcher-window_HOSTA", "HOSTA"))
.unwrap();
// 1. HOSTA pushes to its own folder in the sync dir.
sync_datastores(&a_local, &a_export, true, Some("device-A"), &spec).unwrap();
// 2. HOSTB pulls HOSTA's export. This copy is correct and expected.
sync_datastores(&a_export, &b_local, false, None, &spec).unwrap();
assert!(
bucket_ids(&b_local).contains(&"aw-watcher-window_HOSTA-synced-from-HOSTA".to_string()),
"precondition: HOSTB should hold HOSTA's data as a synced-from-HOSTA bucket, got {:?}",
bucket_ids(&b_local)
);
// 3. HOSTB pushes its own data to the sync folder.
sync_datastores(&b_local, &b_export, true, Some("device-B"), &spec).unwrap();
// 4. HOSTA pulls HOSTB's export.
sync_datastores(&b_export, &a_local, false, None, &spec).unwrap();
(a_local, b_export)
}
/// The fix location: a push must export only buckets that originate on this
/// host. Re-exporting buckets pulled from a peer is what lets data round-trip.
#[test]
fn test_push_does_not_reexport_synced_buckets() {
let (_a_local, b_export) = round_trip();
assert_eq!(
bucket_ids(&b_export),
Vec::<String>::new(),
"HOSTB's export must not contain buckets it merely synced from HOSTA"
);
}
/// The reported symptom: HOSTA must never end up with a `-synced-from-HOSTA`
/// copy of its own bucket, which renders every event twice in /timeline.
#[test]
fn test_own_data_does_not_return_via_peer() {
let (a_local, _b_export) = round_trip();
assert_eq!(
bucket_ids(&a_local),
vec!["aw-watcher-window_HOSTA".to_string()],
"HOSTA must not gain a synced-from-HOSTA copy of its own bucket"
);
}
/// `-synced-from-` is a reserved token in aw-sync's ID grammar, not just a
/// naming convention. `get_or_create_sync_bucket` splits on it to recover the
/// original bucket ID, so a first-hand bucket whose own ID contains it is
/// already mangled on import regardless of provenance filtering.
///
/// This pins that pre-existing behaviour so the reservation is explicit: such a
/// bucket is treated as a synced copy. Issue #649 tracks moving provenance into
/// bucket metadata, which removes the dependency on the ID string.
#[test]
fn test_synced_from_is_a_reserved_id_token() {
let spec = SyncSpec::default();
let local = datastore("reserved-local");
let export = datastore("reserved-export");
// A bucket the sync layer would mangle anyway: the ID grammar reserves the
// token, so this is not a supported first-hand bucket name.
local
.create_bucket(&bucket(
"aw-watcher-window_HOSTA-synced-from-HOSTB",
"HOSTA",
))
.unwrap();
// A normally-named bucket alongside it, to prove the filter is not blanket.
local
.create_bucket(&bucket("aw-watcher-afk_HOSTA", "HOSTA"))
.unwrap();
sync_datastores(&local, &export, true, Some("device-A"), &spec).unwrap();
assert_eq!(
bucket_ids(&export),
vec!["aw-watcher-afk_HOSTA".to_string()],
"the reserved token marks a bucket as synced; unmarked buckets still export"
);
}