DefinePK

Abstract

In today’s rapidly evolving digital landscape, data duplication can significantly impede system efficiency and resource management. An adaptive AI-powered pluggable de-duplication algorithm proposed, designed to dynamically adjust to a wide range of computational environments, including blockchain networks and cloud infrastructures. The algorithm employs machine learning techniques to analyze various environmental factors, such as data size (from 1 GB to 100 GB), processing speed requirements and system architecture (e.g., sharing versus non-sharing environments). Based on this analysis, the algorithm selects the most suitable de-duplication method, ensuring it is both feasible and efficient for the given scenario. The proposed solution can evaluate multiple de-duplication techniques such as hash-based and chunk-based approaches by learning from historical datasets and real-time performance metrics. This allows it to achieve high accuracy and fast processing speeds. Its adaptive and pluggable nature makes it easily customizable for specific infrastructures, optimizing resource usage and ensuring seamless integration across various platforms. The algorithm continuously adjusts its strategy as system demands evolve, enhancing data management efficiency in a wide array of use cases in adaptive learning techniques.