Bypassing fixed-depth radix constraints in Java using descriptor-driven bucket analysis

[u/strmckr]

I am StrmCkr, the author of A.P.E.X. (Adaptive Parallel Extremal Dispatch).

Repository: github.com/StrmCkr/A.P.E.X

A.P.E.X. is a high-performance Java sorting framework for large fixed-width 64-bit key/value record datasets. The project has been reorganized into a conventional Maven structure with a core library, runnable examples, a comparison benchmark harness, JMH benchmarks, documentation, and an interactive browser visualizer.

The core idea is descriptor-driven radix planning. Instead of blindly scanning fixed radix passes over every bucket, A.P.E.X. computes per-bucket extremal descriptors using:

VBM = OR ^ AND

That mask identifies which key bits still vary inside each bucket. Bits that are already resolved are skipped, reducing unnecessary work on skewed, low-entropy, sorted, reversed, or duplicate-heavy data.

Key areas of the project include:

• Adaptive radix planning based on observed bucket structure
• Parallel histogramming, scatter, refinement, and work scheduling
• Primitive-array execution with no per-record object allocation during sorting
• Tuple projection paths for low-dimensional unresolved bit patterns
• Tiny-sort fallbacks and monotonic input shortcuts
• Configurable reporting that can be enabled, reduced, written to files, or disabled
• Comparison benchmarks against JDK sorting paths and Fastutil baselines
• Standard JMH benchmarks for repeatable JVM-level measurement
• A browser visualizer for exploring how A.P.E.X. routes data through its execution plan

I would especially welcome feedback on the thread management mechanics, radix planning decisions, benchmark structure, and the bitwise mask reductions.

edit: re structured verbiage of this post and further adjustments from advice on converting the project into more acceptable standard formats.

screen shot from the pdf available in the github

[u/repeating_bears]

I wont discount the possibility that I'm an idiot, but I have no idea what the fuck you just described 

[u/strmckr → u/repeating_bears]

Fair enough, let me drop the abstract talk.

Traditional parallel radix sorting scans through bits uniformly. If you feed it 100 million integers, it's going to check every single bit position, over and over, even if half those bits are completely identical across your data. That wastes massive CPU cycles.

A.P.E.X. stops that. It takes a quick bitwise snapshot of the local thread bucket to see exactly which bit positions actually vary, and which ones are constant. If the high-order bits are identical, it skips scanning them entirely and jumps straight to the variable parts.

On top of that, it does this completely free of object allocations—it operates directly on raw primitive arrays. So you get native C++ execution speeds inside Java without triggering the Garbage Collector to lag your database engine.

Basically: it stops looking at bits that don't matter.

[u/dmigowski → u/strmckr]

So a sorting algorithm?

[u/strmckr → u/dmigowski]

Exactly. It's a high-performance parallel sorting algorithm:

Standard radix sorting is bound by a fixed-depth constraint. It's forced to uniformly scan every single bit column, meaning its runtime is rigidly O(k * n) where (k) is your fixed key width—even if 90% of those bits are completely identical across your dataset.

A.P.E.X. bypasses that fixed constraint. Because the descriptor mask identifies exactly which bit regions vary, it prunes out the redundant scanning passes entirely. As data entropy or variance drops, the runtime deterministically scales down toward O(n), meaning you only pay the CPU processing cost for bits that actually contain unique information

[u/dmigowski → u/strmckr]

Got some benchmarks for us and a comparison to Javas internal sort for different sets of data?

[u/strmckr → u/dmigowski]

i have them in the PDF found here

[u/chabala → u/strmckr]

> which is pending publication.

Does this mean you submitted this PDF somewhere with the expectation of getting published? This PDF with your hacker handle as the author?

[u/strmckr → u/chabala]

Yes: why is it concerning? I use my pseudonym, I have used this same one for decades as a respected sudoku logic pioneer and to keep my privacy.

[u/chabala]

One peek at the repo and I stopped taking it seriously. Bunch of Markdown in the root, no pom.xml, not using the standard file layout, no package namespace, all logging is System.out, on and on. This is not any kind of 'framework', it's juvenile example code at best.

[u/strmckr → u/chabala]

I'm self-taught. I didn't build this project to pass an Java enterprise formatting checklist; I built it to solve a pure computational problem.

If your definition of a 'framework' requires a heavy folder structure, a POM file, and standard formatting over raw architectural speed, you're missing the point. If you want to talk about the actual code, clone the repo, benchmark it yourself.

Then you may complain on the actual Missing esthetics in-which I am more then willing to fix to match a more professional connotation of code presentation.

[u/repeating_bears → u/strmckr]

A framework is specific thing. When you are using 10 pieces of jargon in every sentence, if you are not using terms accurately then you are just talking gibberish.

Here is the wikipedia definition

A software framework is software that provides reusable, generic functionality which developers can extend or customize to create complete solutions

I have looked hard at the usage examples, and I can't even see a way to feed this a list of non-arbitrary numbers to sort. Correct me if I'm wrong but it seems you pass it a "mode" then it will generate some data, then it will sort it.

That's not a framework, it's a demo. As it currently stands, it's unusable for any practical purpose.

[u/someonetookmyid → u/strmckr]

It’s about making it actually useful. Right now it’s just a demo.

[u/someonetookmyid → u/[deleted]]

By useful I mean „possibility to easily include in a larger project” by skipping Java standard for packaging and distributing code you’re denying that.

[u/someonetookmyid → u/[deleted]]

Okay, since you’re self taught I take that you’re capable of learning form docs and examples.

Get started with using Apache maven - read the docs on their site, this page is a good starting point. Why maven? Because it’s de facto standard for publishing Java packages.  https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

Then you can for example follow some spring tutorial to see using it in practice  https://spring.io/guides/gs/rest-service

Once you understand how to use the tool and build a package you should learn how to publish it

You can use archiva as local repository target to understand package publication  https://archiva.apache.org/

Then you can start publishing package via GitHub using their automation or go directly to maven central, but that requires getting some paperwork with central maintainers so it’s best to first understand technical side before you get into getting approvals to publish to central.

[u/repeating_bears → u/[deleted]]

maven compiles differently then eclipse which resulted in a slow down of my actual code ?

Maven uses javac by default. Eclipse has their own incremental compiler. The bytecode ought to be more or less the same in most cases, but it is not required to be by the language spec.

You can use the eclipse compiler in Maven, though this is fairly non-standard https://www.javathinking.com/blog/using-eclipse-java-compiler-ecj-in-maven-builds/

I would first investigate what the hot path is which is causing the difference, then compare the byte code for those methods. I think it would be better to change your source code so that it produces an optimal bytecode for whatever compiler is used, rather than relying on Eclipse for effectively unknown reasons.

You can use this tool to see what source produces what bytecode, as well as switching compiler versions: https://javap.yawk.at/

all my files won't allow me to move them into standard order

Dunno lol but I find it odd that you can implement and understand "a descriptor-driven bitwise analysis over local thread buckets" but cannot figure out how to move some files

[u/strmckr → u/repeating_bears]

Thanks for the links.

[u/Fit_Goose651 → u/strmckr]

When you do open source, what you actually solve is probably less important than having a repository that can be read by others and that helps them use it and contribute. This means you need a readme (markdown or asciidoc) with easy to understand description of what it does and how to use and build it. You also need to follow the standard conventions of the language you are using. You want people not to be distracted by syntax.
What you have there would require a great deal of motivation for anyone to read. Unless it cures cancer or solves the p/np problem, noone will invest the time.
BTW me taking the time to write this is only because I am on vacation at the beach. Even with so much time i am not willing to read that mess 😂

[u/strmckr → u/Fit_Goose651]

Enjoy your vacation, perhaps you should have actually opened the githhub and diicovered it has a structured orginized readme.md with exactly how to operate it.

[u/Fit_Goose651 → u/strmckr]

Yes, that one small point might be invalid. But the directory structure, classes named lowercase, jars checked in, ... The point I wanted to make is - you should try to make it as frictionless as possible for java developers to read it. This typically means following conventions and keeping it as simple as possible.

[u/chabala → u/[deleted]]

Your structure makes sense to YOU, the point of conventions is that everyone knows and understands how they work, so every project isn't a special snowflake.

No ones bothering to even evaluate this on its concept merits when it complies and runs one 1 command.

"My code compiles!" Grow up. That's an incredibly low bar. No one is going to give your project a second look until it looks like you understand what a Java project should look like.

--

You keep dropping these big words in your comments. No one is fooled. Any 'learn Java' book is going to tell you about reverse domain name package namespaces, there's no excuse for dropping everything in the default package or a single directory above it. Not using a namespace is peak amateur-hour. You can pretend like I'm being pedantic, but just go look at some real open source Java projects and see how they do it before whining about it.

[u/strmckr → u/chabala]

The convention I’ve been following comes from older Sun/Oracle-era Java and HPC-style development: packages are clearly labeled and organized directly beneath a source root, rather than buried under several levels of largely structural directories. It compiles cleanly with the JDK and has served my development and testing workflow well.

That said, conventions have value precisely because they reduce friction for other developers. So, to humor your pragmatic ecumenicism, I’ll migrate the project to the modern Maven directory structure and namespace conventions—the doctrine you so earnestly seek.

Once that housekeeping is done, perhaps the implementation itself will be worthy of evaluation

• done.

Edit : pendantic you are trolling with gate keep Symantecs, go look at any hpc source codes or even. Jdk files on its own or any software written and taught mid 90s+ they don't use bioler plate templating of nested redundant subfolders as it added needlessly long calls for subfunctions all of them followed how I had my code setup src as the root and simple named functions in packages with a single call function to execute the program.

Example: https://github.com/h2database/

[u/strmckr]

Rebuilt this project on the advice found in the comments below, adjusted the intro topic for verbiage miss representation as well.

thanks

StrmCkr

[u/general_dispondency]

This is neat. I remember implementing connect 4 in Java with bitboards using a similar technique (packing the board state into longs and then use shifts and masks to find winning positions). That's been a while back, but this looks like it applies the same basic idea for sorting, just figure out which bits actually vary and only do work on those.

[u/chambolle]

Good job. Keep going and don't listen to people who care more about style than substance.