Skip to content

[WIP] Test llap io cache disabled mode #6537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tanishq-chugh wants to merge 5 commits into
base: master
Choose a base branch
from
+5 −3
Draft

[WIP] Test llap io cache disabled mode #6537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8480a04
[WIP] Test llap io disabled mode
tanishq-chugh Jun 12, 2026
ef85e8d
Revert "[WIP] Test llap io disabled mode"
tanishq-chugh Jun 12, 2026
6d71167
Update in HiveConf for better coverage
tanishq-chugh Jun 12, 2026
2137e4d
Test UTs after ORC Reader NPE fix
tanishq-chugh Jun 13, 2026
cea1bf6
Disable delete deltas io cache
tanishq-chugh Jun 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4890,7 +4890,7 @@ public static enum ConfVars {
LLAP_IO_NONVECTOR_WRAPPER_ENABLED("hive.llap.io.nonvector.wrapper.enabled", true,
"Whether the LLAP IO layer is enabled for non-vectorized queries that read inputs\n" +
"that can be vectorized"),
LLAP_IO_MEMORY_MODE("hive.llap.io.memory.mode", "cache",
LLAP_IO_MEMORY_MODE("hive.llap.io.memory.mode", "none",
new StringSet("cache", "none"),
"LLAP IO memory usage; 'cache' (the default) uses data and metadata cache with a\n" +
"custom off-heap allocator, 'none' doesn't use either (this mode may result in\n" +
Expand Down Expand Up @@ -5020,7 +5020,7 @@ public static enum ConfVars {
"notifications are received by the daemon. Sweep phase of proactive eviction will only do the cache policy " +
"cleanup in this case. This can increase cache hit ratio but might scale bad in a workload that generates " +
"many proactive eviction events."),
LLAP_IO_CACHE_DELETEDELTAS("hive.llap.io.cache.deletedeltas", "all", new StringSet("none", "metadata", "all"),
LLAP_IO_CACHE_DELETEDELTAS("hive.llap.io.cache.deletedeltas", "none", new StringSet("none", "metadata", "all"),
"When set to 'all' queries that use LLAP IO for execution will also access delete delta files via " +
"LLAP IO layer and thus they will be fully cached. When set to 'metadata', only the tail of delete deltas " +
"will be cached. If set to 'none', only the base files and insert deltas will be channeled through LLAP, " +
Expand Down
4 changes: 3 additions & 1 deletion llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,9 @@ private void performDataRead() throws IOException {
// TODO: I/O threadpool could be here - one thread per stripe; for now, linear.
boolean hasFileId = this.fileKey != null;
OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileKey, -1, 0) : null;
pathCache.touch(fileKey, split.getPath().toUri().toString());
if (pathCache != null) {
pathCache.touch(fileKey, split.getPath().toUri().toString());
}
for (int stripeIxMod = 0; stripeIxMod < stripeRgs.length; ++stripeIxMod) {
if (processStop()) {
return;
Expand Down
Loading