Distributed file system analytics often depends on a practical trade-off: organize data to accelerate the queries that matter, or preserve a layout that is straightforward to replicate and recover. INSTalytics, a research system developed for large-scale analytics, explores a way to address both needs. It co-designs the storage and compute layers so a file can support several partitioning dimensions without the extra full copies that conventional approaches require.
The idea is relevant to teams running scan-intensive workloads across large clusters. Filters can skip data when it is organized by the queried column, while joins can avoid some network movement when matching data is colocated. The challenge is making those benefits available across more than one query dimension without compromising availability or making failure recovery prohibitively expensive.
Why a single partitioning dimension is limiting
In many analytics systems, records are physically organized by one column. A query filtering on that column may read only the relevant partitions rather than scan the full file. Similarly, if two files are consistently partitioned on a join key, corresponding partitions can be processed together, reducing the need for a cluster-wide shuffle.
That layout is less useful when queries filter or join on other columns. Those operations may need to scan much more data or move records across the network. Maintaining separately organized full copies can address the limitation, but increases storage use and the write work needed to keep copies current.
INSTalytics was motivated by analysis of a production workload reported in the paper: one partitioning dimension covered about 33% of files, while four dimensions covered about 83%. These figures describe that workload, not a universal expectation, but illustrate why broader partition support can matter.
How logical replication supports multiple dimensions
Conventional three-way replication stores byte-identical copies to protect against failures. INSTalytics instead uses logical replication: each replica contains the same records, but organizes them around a different partitioning dimension. The storage footprint remains based on three replicas, while queries can use whichever layout fits their filter or join.
Simply rearranging each entire file independently would make recovery costly. A block in one ordering could be scattered across most of the file in another, so rebuilding a failed block might require reading a very large amount of data. INSTalytics addresses this with a layout that limits how far records can be dispersed.
Super-extents and intra-extent buckets
The system groups contiguous extents into super-extents and performs partitioning within each group. This keeps related data within a bounded region rather than distributing it across the whole file. Within extents, records are further arranged into buckets using other dimensions. During recovery, the system can gather the needed portions from corresponding buckets instead of reading every full extent in the group.
In the paperâs example configuration, a super-extent contains 100 extents. Chained buckets allow recovery to read small portions from those extents, bringing the aggregate recovery read volume close to that of recovering from a conventional physical replica. The exact layout is configurable, so the size of a super-extent represents a balance between partitioning granularity and recovery considerations.
The arrangement also preserves the fileâs coarse ordering across super-extent boundaries. That ordering can provide a fourth filtering dimension: a query may eliminate whole super-extents based on the original layout, alongside the three dimensions represented by logical replicas. This additional dimension supports filtering, but does not provide the same join-colocation benefit as the first three.
Availability and recovery remain design requirements
Equalizing recovery read volume does not by itself guarantee equal availability. With physical replicas, a block remains available if at least one of its copies survives. Under logical replication, a missing extent may need data drawn from multiple locations, creating a different failure exposure.
INSTalytics addresses this with parity extents and fault-isolated placement. XOR parity can tolerate a failure within a parity group, while placement separates the logical replicas of a super-extent across failure domains. The design therefore considers both independent machine failures and correlated failures, such as those affecting a rack. The appropriate configuration depends on the systemâs failure assumptions and operational requirements; the paperâs results should not be treated as a guarantee for every cluster.
Reducing join movement with sliced reads
When two files share a logical partitioning dimension, corresponding partitions can be joined locally rather than sent through a global shuffle. This localized shuffle reduces the number of inputs meeting at each join task and can reduce cross-rack traffic when placement is aligned. It does not necessarily eliminate intermediate disk writes and reads.
For that, INSTalytics adds a sliced-read API. A compute node requests a semantic slice of an extentâfor example, a narrower range within a partitionâso work on a large join can be divided into stages. Because multiple compute nodes may request slices from the same extent, a storage node coordinates requests, reads and reorganizes the extent, then serves the requested portions. It selectively caches data needed by late-arriving requests rather than retaining the entire extent.
This approach requires coordination between storage and compute, including scheduling staged work and aligning file partitions. It also relies on the storage layer understanding supported file formats through adapters that translate between stored extents and records. The design preserves a block-oriented interface for clients, but logical replication is limited to formats with suitable adapters and metadata.
Practical implications for analytics infrastructure
In a 500-server evaluation, the authors reported recovery disk and network load within 10% of physical recovery. Their benchmarks and selected production queries also showed that query benefits varied with workload: filters and joins on supported dimensions benefited, while queries that filtered little data and did not join showed no meaningful gains. These results demonstrate the potential of the approach, not a blanket performance outcome for other environments.
For engineering and operations teams, the broader lesson is to treat storage layout as part of query execution rather than a fixed backend decision. Assessing this kind of design requires attention to:
- Which columns are commonly used for filters and joins, and whether those patterns are stable.
- Whether data formats can be decoded and reorganized safely at the storage layer.
- How partition skew, recovery traffic, and failure-domain placement will be managed.
- Whether the compute scheduler and query optimizer can select dimensions and coordinate staged reads.
- Whether the read-to-write profile justifies the work of creating logical replicas.
Conclusion
INSTalytics shows how co-design between a distributed file system and analytics engine can extend partitioning benefits beyond a single column. Logical replicas, super-extents, chained buckets, and coordinated sliced reads work together to support filtering and joins while keeping recovery costs near those of physical replication in the reported evaluation. The practical value depends on workload fit and careful integration, but the central principle is broadly useful: storage organization, query planning, and failure recovery should be designed as connected parts of the analytics system.
Access Official Technical PDF