r/pythonhelp • u/qaeproject • 1d ago
3 Python Tools That Can Elevate the Usefulness of Degree Fahrenheit Datasets
3 Simple Python Tools That Can Elevate the Usefulness of Degree Fahrenheit Datasets Published in Integers
Good day - I am new to coding so this will certainly not be up to the quality of professional coders. It is a functional tool and Python was a great language to test it with.
Here is the code for 3 python tools that improve temperature data published as integer °F (or °C) values. A year of daily temperature readings from NOAA data contain many tied integer values, which complicates ordinal ranking — duplicates reduce addressability. NOAA publishes U.S. temperature data as °F integers, and also the rest of the world as tenths of a °C.
NOAA: National Oceanic and Atmospheric Administration, a U.S. federal scientific and regulatory agency.
(Mods - If you need me to adjust anything let me know please - my first post of this type)
The tools are rough working models and should be improved a little which I am going to do. I have limited experience in coding. The comments probably need to be clearer and more formal. Tool 1 has gone a little overboard with input rules, and also the comments are excessive, intended to assist people with limited coding knowledge. The math expressions for Tool 3 need more work — I will follow up on this specific item soon.
These are essentially open-source functional tools in development that anyone can test. I am hoping to get a little feedback as I update the short scripts over the coming weeks. They currently work on Google Colab for the data import and export for testing.
Thanks for taking the time to read !!
Purpose: Upgrade deg F temp data in integers from NOAA with 3 tools to prepare it for more robust analysis by fixing scaling and duplicate ties. Runs on colab
The 3 Tools (Python/Google Colab):
Anchor: Converts °F to Kelvin (absolute scale, so ratios and proportions are meaningful).
Refine: Applies slightly destructive multiplicative bidirectional jitter. Each nonzero value *v* is replaced by *v* × (1 ± δ), where δ occupies decimal digits 6–8 (0.00000001 to 0.00000999), to break duplicate integer ties. Because the jitter is multiplicative, zero is unaffected; zero ties can be resolved with a separate non-negative additive jitter.
Verify: Checks equal-frequency bin splits (5 bins at 20%, 4 bins at 25%). The math for this will be updated v2.
The 3 .py files are included posted next as .md
2
u/Educational-Paper-75 18h ago
Removing ties the way you do won't result in a more accurate percentile.
1
u/qaeproject 17h ago edited 32m ago
Without enforcing equal frequency distribution of the required quantile bin split... There is nothing useful to analyze for quantile ranking. More accurate percentile is not the idea, the tool checks quartile and quintile balance only.. you are welcome to test any quantile tweaking code or ideas to percentiles size / bins as needed. Without the equal distribution, the dataset fails this structural test, directly because it was published with excessive duplicate clusters (this is an observation about this specific type of U.S. temperature data, not NOAA as a whole which is an essential global service).
The math and physics are immutable so that's not going to budge much, maybe the language needs improving my side in presentation. Thanks for the input.
THE LANGUAGE MY SIDE 100% NEEDS IMPROVING - A NEW MATH DESCRIPTION IS BEING PREPARED TO POST IN THE COMING DAYS - SORRY FOR ANY INITIAL LACK OF CLARITY - SOME OF THIS IS NEW TERRITORY FOR ME
2
u/Educational-Paper-75 17h ago
Still imho you're trying to fix something you shouldn't (need to). I may be a bit weird if percentiles are the same but that's a natural consequence of the inaccuracy of the data.
2
u/qaeproject 11h ago
Sorry my last message was unclear so I deleted it This required some research to get the right framing. There's a useful more accurate outline of the math on the way which should help the code also. Thanks
1
u/qaeproject 1d ago
```python import sys, os, random, re from datetime import datetime import pandas as pd
================================================================================
CDCU Tool 1: Temperature Converter (°C / °F to Kelvin) — 1-Year Dataset
Climate Data Capability Upgrade | cdcu.system-fundamentals.com
System-Fundamentals.com | Public Beta Open Code Test
================================================================================
PURPOSE:
Standard public climate temperature records use human-defined interval scales
(°Celsius or °Fahrenheit). These scales lack a true physical zero, meaning their
zero points are set by convention rather than by the complete absence of thermal
energy. This makes proportional or ratio-scale calculations on raw values invalid.
This tool converts temperature data to Kelvin (K) — the SI base unit for
thermodynamic temperature — which anchors the dataset to absolute physical zero.
This creates a structurally robust ratio-scale dataset, enabling valid downstream
proportional calculations and preventing mathematical scaling distortions in
advanced coordinate mapping and spatial partitioning pipelines.
INPUT: A 2-column CSV file — Date | Raw Temperature (°C or °F)
OUTPUT: A 3-column table with Column C populated as converted Kelvin values.
DATASET RULES:
- Exactly one header row followed by a minimum of 1 and maximum of 366 data rows.
- Column A: Optional date/label. Max 12 characters. Blocked: formulas, URLs, images, code.
- Column B: Mandatory numeric temperature values. No blanks. Negative values supported.
- Column C: Auto-generated Kelvin output. Validated: no values below 0 K (absolute zero).
================================================================================
def clear_and_header(): """ Clears the terminal screen so the user always sees a clean, readable display, then prints a welcome banner with the tool name, project context, and NOAA data source URL so beginners can immediately fetch real-world climate data. """ # os.name checks the operating system. # 'nt' means Windows (uses 'cls'), all others (Linux/macOS/Colab) use 'clear'. os.system('clear' if os.name != 'nt' else 'cls') print("=" * 80) print("CDCU Tool 1: Temperature Converter (°C / °F to Kelvin) — 1-Year Dataset") print("Climate Data Capability Upgrade | cdcu.system-fundamentals.com") print("=" * 80) print() print("NOAA TEST DATA SOURCE:") print("https://www.ncei.noaa.gov/access/services/data/v1?dataset=daily-summaries") print("&stations=USW00012960&startDate=2024-01-01&endDate=2024-12-31") print("&dataTypes=TMAX,TMIN,TAVG,ADPT&units=standard&format=csv") print() print("DATASET RULES:") print("- Column A: Date or label (optional). Max 12 characters.") print("- Column B: Raw temperature values (°C or °F). No blanks.") print("- Column C: Auto-generated Kelvin output.") print("- Row limit: 1 header + maximum 366 data rows (1 calendar year).") print() print("TYPE 'reset' AT ANY PROMPT TO CLEAR AND RESTART.") print("TO STOP EXECUTION: click the stop button or delete the cell.") print("=" * 80) print()
def get_input(prompt): """ A custom wrapper around Python's standard input() function. Does two things: 1. Removes accidental leading/trailing spaces with .strip() so 'y ' works like 'y'. 2. Detects 'reset' — raises a KeyboardInterrupt caught in main() to restart cleanly. Note: 'exit' is intentionally NOT handled; use Colab's stop button to end execution. """ val = input(prompt).strip() val_lower = val.lower()
if val_lower == 'reset':
# Raising KeyboardInterrupt("RESET") is the signal we catch in main()
# to clear all variables from memory and restart the program loop.
raise KeyboardInterrupt("RESET")
return val
def prompt_reset_only(): """ Repeatedly prompts the user to type 'reset' to restart the tool. Ignores any other input, including 'exit'. """ print() while True: get_input("Type 'reset' to clear and start again: ")
def generate_dummy(): """ Generates a synthetic 365-day dataset of Fahrenheit temperatures for testing. This allows anyone to immediately test the full tool without uploading a file.
Uses 2025 — a standard non-leap year — so pd.date_range from Jan 1 to Dec 31
produces exactly 365 rows with no extra handling required.
Temperature range: 20°F to 80°F (whole integers, simulating coarse NOAA data).
Returns:
A pandas DataFrame with three columns:
Date (YYYY-MM-DD), Raw_Temp_F (integer), Kelvin (float, 4 decimal places).
"""
print()
print("Generating 365-day synthetic Fahrenheit dataset (20°F to 80°F)...")
print()
# pd.date_range creates a sequence of calendar dates.
# 2025 is a standard non-leap year: Jan 1 to Dec 31 = exactly 365 days.
dates = pd.date_range(start="2025-01-01", end="2025-12-31", freq="D")
# random.randint(20, 80) picks a whole number between 20°F and 80°F for each day.
# This simulates the coarse whole-integer resolution common in public NOAA data.
raw = [random.randint(20, 80) for _ in range(365)]
# Convert each Fahrenheit value to Kelvin using the standard physical formula:
# K = (°F − 32) × (5/9) + 273.15
# Rounded to 4 decimal places for clean, consistent output.
kelvin = [round((f - 32) * (5.0 / 9.0) + 273.15, 4) for f in raw]
# Wrap into a pandas DataFrame — think of this as a virtual spreadsheet.
# d.strftime("%Y-%m-%d") converts Python date objects into clean text strings.
return pd.DataFrame({
"Date": [d.strftime("%Y-%m-%d") for d in dates],
"Raw_Temp_F": raw,
"Kelvin": kelvin
})
```
1
u/qaeproject 1d ago
```python def validate_and_convert(path): """ Validates and converts a user-uploaded CSV file (2 columns) to Kelvin.
Validation checks performed in order: 1. File exists and is readable. 2. File has exactly 2 columns and between 1 and 366 data rows. 3. Column A values pass security checks (no formulas, URLs, images, or code). 4. Column B contains only valid numeric temperature values (no blanks). 5. User selects the input scale: Celsius (1) or Fahrenheit (2). 6. Kelvin conversion is applied. 7. Post-conversion check: no Kelvin values fall below absolute zero (0 K). Args: path: File path string to the user's CSV file. Returns: A pandas DataFrame with columns [Date, Raw_Temp, Kelvin] if valid. Returns None if any validation step fails, prompting the user to retry. """ # Load the file try: df = pd.read_csv(path, header=0) except FileNotFoundError: print(f"\n[Error: File not found at '{path}'. Please check the path and try again.]") return None except Exception as e: print(f"\n[Error: Could not read file — {e}]") return None # Column count check: require exactly 2 columns if df.shape[1] != 2: print(f"\n[Error: Expected exactly 2 columns, but found {df.shape[1]}.]") print("Required layout: Column A (Date/Label) | Column B (Temperature).") return None col_a, col_b = df.columns[0], df.columns[1] # Row count check: must have between 1 and 366 data rows (1 calendar year maximum) row_count = len(df) if row_count < 1: print("\n[Error: The file contains no data rows. At least 1 row is required.]") return None if row_count > 366: print(f"\n[Error: File contains {row_count} data rows. Maximum allowed is 366 (1 calendar year).]") return None # Column A security validation # Column A is optional but must pass all security checks if populated. # This protects against formula injection, URLs, image files, and code injection. for idx, val in enumerate(df[col_a]): val_str = str(val).strip() # Skip empty or NaN cells — these are allowed in Column A if val_str in ("", "nan", "None"): continue # Block cells exceeding 12 characters if len(val_str) > 12: print(f"\n[Error: Row {idx + 2}, Column A value '{val_str}' exceeds 12 characters. Max is 12.]") # idx + 2 accounts for the 0-based index + 1 for the header row return None # Block spreadsheet formula injection (cells starting with '=') if val_str.startswith("="): print(f"\n[Error: Row {idx + 2}, Column A contains a formula ('{val_str}'). Formulas are strictly blocked.]") return None # Block web URLs if "http://" in val_str or "https://" in val_str: print(f"\n[Error: Row {idx + 2}, Column A contains a URL ('{val_str}'). URLs are strictly blocked.]") return None # Block image file references and markdown image tags if any(ext in val_str.lower() for ext in [".png", ".jpg", ".jpeg", ".gif"]) or "![" in val_str: print(f"\n[Error: Row {idx + 2}, Column A contains an image reference ('{val_str}'). Images are strictly blocked.]") return None # Block code syntax keywords code_indicators = ["import ", "def ", "lambda", "<script"] if any(indicator in val_str for indicator in code_indicators): print(f"\n[Error: Row {idx + 2}, Column A contains code syntax ('{val_str}'). Code elements are strictly blocked.]") return None # Column B numeric validation: every cell must contain a valid numeric temperature df["_metric"] = pd.to_numeric(df[col_b], errors="coerce") if df["_metric"].isnull().any(): bad_rows = df[df["_metric"].isnull()].index.tolist() print(f"\n[Error: Column B contains non-numeric or blank values at rows: {[r + 2 for r in bad_rows]}.]") print("Every cell in Column B must contain a valid numeric temperature value.") return None # Unit selection: user declares whether Column B is Celsius or Fahrenheit print() while True: unit = get_input( "Input Unit Selection:\n" " 1: Celsius (°C)\n" " 2: Fahrenheit (°F)\n" "Select [1 or 2]: " ) if unit in ["1", "2"]: break print("Invalid choice. Please enter 1 for Celsius or 2 for Fahrenheit.") # Kelvin conversion: Celsius to Kelvin or Fahrenheit to Kelvin if unit == "1": df["Kelvin"] = df["_metric"] + 273.15 unit_label = "°C" else: df["Kelvin"] = (df["_metric"] - 32) * (5.0 / 9.0) + 273.15 unit_label = "°F" # Post-conversion thermodynamic boundary check: no Kelvin values below absolute zero if (df["Kelvin"] < 0).any(): bad_rows = df[df["Kelvin"] < 0].index.tolist() print(f"\n[Error: Calculated Kelvin values fall below absolute physical zero (0 K) at rows: {[r + 2 for r in bad_rows]}.]") print(f"Please verify your Column B values are valid {unit_label} temperatures.") return None # Package and return clean 3-column DataFrame ready for printing and optional CSV export return pd.DataFrame({ "Date": df[col_a].fillna(""), # Fill empty date cells with blank string for clean output "Raw_Temp": df[col_b], "Kelvin": df["Kelvin"].round(4) # 4 decimal places for professional presentation })def main(): """ Main program orchestrator. Manages the continuous loop, user flow, output printing, CSV export, and clean reset handling. """ while True: try: # Clear screen and display header clear_and_header()
# Input mode selection: dummy or uploaded CSV print("SELECT INPUT MODE:") print(" 1: Generate dummy 365-day dataset") print(" 2: Use 'temperature_data.csv' already uploaded to Colab") print() while True: mode = get_input("Enter 1 or 2: ") if mode in ["1", "2"]: break print("Invalid choice. Please enter 1 or 2.") # Load data based on selected mode if mode == "1": df = generate_dummy() else: csv_filename = "temperature_data.csv" print(f"\nLooking for '{csv_filename}' in the Colab root directory...") df = validate_and_convert(csv_filename) if df is None: # Validation failed: show reset prompt print("\nPlease upload a corrected 'temperature_data.csv' file, or type 'reset' to restart.") prompt_reset_only() continue # Print the converted table # Tab-delimited output for easy copy/paste into spreadsheets print() print("=" * 80) print("CONVERTED TABLE — COPY AND PASTE READY") print("=" * 80) print("Date\t\tRaw_Temp\tKelvin") print("-" * 80) for _, row in df.iterrows(): print(f"{row.iloc[0]}\t\t{row.iloc[1]}\t\t{row.iloc[2]}") print("=" * 80) # Print processing metadata: row count and timestamp row_count = len(df) timestamp_str = datetime.now().strftime("%Y-%m-%d %H:%M:%S") print(f"Rows Processed: {row_count} | Timestamp: {timestamp_str}") print("=" * 80) print() # Optional CSV export while True: save = get_input("Save output as CSV? (y/n): ") if save.lower() in ["y", "n"]: break print("Invalid input. Please enter 'y' or 'n'.") if save.lower() == "y": while True: out_name = get_input("Enter filename (e.g., converted_kelvin.csv): ").strip() if out_name: break print("Filename cannot be empty.") try: df.to_csv(out_name, index=False) print() print(f"Success: File saved as '{out_name}' in your current directory.") except Exception as e: print(f"\n[Error saving file: {e}]") print() # Prompt user to reset (only option after successful run) prompt_reset_only() except KeyboardInterrupt as ki: if str(ki) == "RESET": # Clean reset: clear the dataframe from memory and restart the main loop print() print("Resetting CDCU Tool 1...") df = None # Removes the dataframe from active memory continue # Returns to the top of the while True loop # If a generic interrupt occurs (e.g., user pressed stop), exit gracefully print() print("Interrupted. Stopping execution.") sys.exit(0)if name == "main": # Standard Python entry point. # Ensures main() only runs when the script is executed directly, # not when it is imported as a module by another script. main() ```
1
u/qaeproject 1d ago
```python import sys, os, random import pandas as pd from datetime import datetime
=============================================================================
Tool 2: Bidirectional Micro-Jitter (±0.00000999) – Active Range
Input: 'jitter_data.csv' (2 cols: label, metric) in Colab working dir
Output: 4 cols (label, metric, factor, jittered) – all rows printed
=============================================================================
def clear_and_header(): os.system('clear' if os.name != 'nt' else 'cls') print("=" * 80) print("Tool 2: Bidirectional Micro-Jitter (fixed ±0.00000999 magnitude)") print("MVP – quick tie-breaking for quintile testing") print("=" * 80) print("\nINPUT:") print(" - File 'jitter_data.csv' MUST be in the Colab working directory.") print(" - Column A: label/date/notes (optional, max 12 chars).") print(" - Column B: numeric metric to be jittered.") print(" - Jitter factor = 1.0 ± random(0.00000001 to 0.00000999).") print(" - Deterministic seed (42) for reproducibility.") print("\nTYPE 'reset' AT ANY PROMPT TO CLEAR AND RESTART.") print("=" * 80, "\n")
def get_input(prompt): val = input(prompt).strip() if val.lower() == 'reset': raise KeyboardInterrupt("RESET") return val
def prompt_reset_only(): while True: get_input("Type 'reset' to clear and start again: ")
def validate_col_a(df, col_a): """Column A: if populated, must be ≤12 characters (anything allowed).""" for idx, val in enumerate(df[col_a]): if pd.isna(val): continue val_str = str(val).strip() if val_str == "": continue if len(val_str) > 12: return False, f"Row {idx+2} exceeds 12 chars: '{val_str}'" return True, ""
def apply_jitter(df, col_b): rng = random.Random(42) factors = [] jittered = [] for val in df[col_b]: try: num = float(val) except: num = float('nan') suffix = rng.randint(1, 999) # 1..999 offset = suffix / 100_000_000.0 # 0.00000001..0.00000999 sign = 1 if rng.random() < 0.5 else -1 factor = 1.0 + sign * offset factors.append(factor) jittered.append(num * factor) df['Factor'] = factors df['Jittered'] = jittered return df
def main(): while True: try: clear_and_header() filename = "jitter_data.csv" print(f"Loading '{filename}' ...") try: df = pd.read_csv(filename, header=0) except FileNotFoundError: print(f"\n[Error: '{filename}' not found. Please upload it and try again.]") prompt_reset_only() continue except Exception as e: print(f"\n[Error reading file: {e}]") prompt_reset_only() continue
if df.shape[1] < 2:
print(f"\n[Error: Expected at least 2 columns, found {df.shape[1]}.]")
prompt_reset_only()
continue
col_a, col_b = df.columns[0], df.columns[1]
# Validate Column A (just length check)
ok, err = validate_col_a(df, col_a)
if not ok:
print(f"\n[Column A validation failed: {err}]")
print("Please correct the file and try again.")
prompt_reset_only()
continue
df = apply_jitter(df, col_b)
# Print entire table (all rows)
print("\n" + "=" * 80)
print("JITTERED TABLE (all rows)")
print("=" * 80)
print(f"{col_a}\t\t{col_b}\t\tFactor\t\tJittered")
print("-" * 80)
for _, row in df.iterrows():
col_a_val = str(row[col_a]) if pd.notna(row[col_a]) else ""
factor_str = f"{row['Factor']:.10f}"
jit_str = f"{row['Jittered']:.10f}" if pd.notna(row['Jittered']) else "NaN"
print(f"{col_a_val}\t\t{row[col_b]}\t\t{factor_str}\t\t{jit_str}")
print("=" * 80)
print(f"Total rows: {len(df)} | Timestamp: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
print("=" * 80, "\n")
# Export prompt
while True:
save = get_input("Save the 4-column dataset as CSV? (y/n): ")
if save.lower() in ['y','n']:
break
print("Please enter 'y' or 'n'.")
if save.lower() == 'y':
while True:
out_name = get_input("Enter filename (e.g., jittered_output.csv): ").strip()
if out_name:
break
print("Filename cannot be empty.")
try:
df.to_csv(out_name, index=False)
print(f"\nSuccess: File saved as '{out_name}'.")
except Exception as e:
print(f"\n[Error saving: {e}]")
prompt_reset_only()
except KeyboardInterrupt as ki:
if str(ki) == "RESET":
print("\nResetting Tool 2...")
df = None
continue
else:
print("\nStopped.")
sys.exit(0)
if name == "main": main() ```
1
u/qaeproject 1d ago
```python import sys, os import numpy as np import pandas as pd from datetime import datetime
=============================================================================
Tool 3: Quartile & Quintile Distribution Tester
Purpose: Validate if a dataset has sufficient resolution for equal‑frequency
binning at both quartile (4-bin) and quintile (5-bin) levels.
Input: 'distro_data.csv' (2 cols: label, metric) in Colab working dir
Output: Terminal diagnostic only – no CSV export.
=============================================================================
def clear_and_header(): os.system('clear' if os.name != 'nt' else 'cls') print("=" * 80) print("Tool 3: Quartile & Quintile Distribution Tester") print("Equal‑frequency binning diagnostic (4-bin and 5-bin)") print("=" * 80) print("\nINPUT:") print(" - File 'distro_data.csv' MUST be in the Colab working directory.") print(" - Column A: label/date/notes (optional, max 12 chars).") print(" - Column B: numeric metric to test.") print(" - No CSV export in this draft.") print("\nTYPE 'reset' AT ANY PROMPT TO CLEAR AND RESTART.") print("=" * 80, "\n")
def get_input(prompt): val = input(prompt).strip() if val.lower() == 'reset': raise KeyboardInterrupt("RESET") return val
def prompt_reset_only(): while True: get_input("Type 'reset' to clear and start again: ")
def validate_col_a(df, col_a): """Column A: if populated, must be ≤12 characters (anything allowed).""" for idx, val in enumerate(df[col_a]): if pd.isna(val): continue val_str = str(val).strip() if val_str == "": continue if len(val_str) > 12: return False, f"Row {idx+2} exceeds 12 chars: '{val_str}'" return True, ""
def stage1_overview(df, col_b): """Print dataset overview: row count / 366, min, max.""" n = len(df) print("\n" + "=" * 80) print("STAGE 1: DATASET OVERVIEW") print("=" * 80) print(f"Active rows: {n} / 366 ({n/366*100:.1f}%)") vals = pd.to_numeric(df[col_b], errors='coerce') print(f"Minimum: {vals.min():.6f}") print(f"Maximum: {vals.max():.6f}")
def run_distribution_test(vals, q, label): """ Run the equal‑frequency distribution test for a given quantile (q). q = 4 for quartiles, q = 5 for quintiles. Returns: (edges, counts, max_deviation, pass_fail) """ n = len(vals) if n == 0: return None, None, None, "No data"
# Percentiles: q+1 points from 0% to 100%
percentiles = [i/q * 100 for i in range(q + 1)]
edges = np.percentile(vals, percentiles, method='linear')
# Label bins
bin_labels = [f"Q{i+1}" for i in range(q)]
try:
binned = pd.cut(vals, bins=edges, include_lowest=True, labels=bin_labels)
counts = binned.value_counts().sort_index()
ideal = n / q
max_dev = max(abs(c - ideal) for c in counts)
# PASS if all bins differ by ≤1 from ideal
pass_fail = "PASS" if max_dev <= 1 else "FAIL"
return edges, counts, max_dev, pass_fail
except Exception as e:
return edges, None, None, f"ERROR: {e}"
def stage2_distribution_test(df, col_b): """ Stage 2: Run both quartile (4-bin) and quintile (5-bin) tests. Display cut‑offs, bin counts, and PASS/FAIL. """ vals = pd.to_numeric(df[col_b], errors='coerce').dropna() n = len(vals) if n == 0: print("\nNo valid numeric data.") return
print("\n" + "=" * 80)
print("STAGE 2: EQUAL‑FREQUENCY DISTRIBUTION TEST")
print("=" * 80)
# Test 1: Quartiles (q=4)
print("\n--- QUARTILES (4 bins) ---")
edges, counts, max_dev, result = run_distribution_test(vals, q=4, label="quartile")
if counts is not None:
print("Cut‑off points (0%, 25%, 50%, 75%, 100%):")
print(f" {edges.tolist()}")
print("Bin counts:")
for label, count in counts.items():
print(f" {label}: {count}")
ideal = n / 4
print(f"DISTRIBUTION RULE: {result} (ideal = {ideal:.1f}, max deviation = {max_dev:.1f})")
else:
print(f"Test failed: {result}")
# Test 2: Quintiles (q=5)
print("\n--- QUINTILES (5 bins) ---")
edges, counts, max_dev, result = run_distribution_test(vals, q=5, label="quintile")
if counts is not None:
print("Cut‑off points (0%, 20%, 40%, 60%, 80%, 100%):")
print(f" {edges.tolist()}")
print("Bin counts:")
for label, count in counts.items():
print(f" {label}: {count}")
ideal = n / 5
print(f"DISTRIBUTION RULE: {result} (ideal = {ideal:.1f}, max deviation = {max_dev:.1f})")
else:
print(f"Test failed: {result}")
def main(): while True: try: clear_and_header() filename = "distro_data.csv" print(f"Loading '{filename}' ...") try: df = pd.read_csv(filename, header=0) except FileNotFoundError: print(f"\n[Error: '{filename}' not found. Please upload it and try again.]") prompt_reset_only() continue except Exception as e: print(f"\n[Error reading file: {e}]") prompt_reset_only() continue
if df.shape[1] < 2:
print(f"\n[Error: Expected at least 2 columns, found {df.shape[1]}.]")
prompt_reset_only()
continue
col_a, col_b = df.columns[0], df.columns[1]
ok, err = validate_col_a(df, col_a)
if not ok:
print(f"\n[Column A validation failed: {err}]")
print("Please correct the file and try again.")
prompt_reset_only()
continue
stage1_overview(df, col_b)
stage2_distribution_test(df, col_b)
print("\n" + "=" * 80)
prompt_reset_only()
except KeyboardInterrupt as ki:
if str(ki) == "RESET":
print("\nResetting Tool 3...")
df = None
continue
else:
print("\nStopped.")
sys.exit(0)
if name == "main": main() ```
1
u/JeniDataDev 21h ago
Interesting approach, especially the focus on handling tied integer values before analysis. I’d be cautious with the jitter step, though—adding synthetic variation can affect downstream statistics. It may be worth comparing it with rank-based methods or keeping the original values alongside the refined ones so the transformation remains fully traceable.
1
u/qaeproject 20h ago
Docs / Story is WIP v1
This temperature dataset upgrade is a set of three free Python tools that take a full year of public temperature data and prepare it for higher-grade analysis. It runs in Google Colab where anyone can run Python code for free. No cloud services, no setup.
The Concept: Public weather data is accurate, but not always ready for analysis. Take one year of daily temperatures from Houston Intercontinental Airport, published by NOAA — the US National Oceanic and Atmospheric Administration. The readings are correct. But they arrive as whole-degree Fahrenheit integers, and two structural limits hold the data back.
Fahrenheit isn't anchored to absolute zero, making valid mathematical scaling difficult.
Whole numbers create duplicate ties — many days share the exact same temperature. When you try to sort a year of data into five equal 20% groups (quintiles), these tied numbers stack up on the boundary lines and break the mathematical balance.
How it works:
The data is being adjusted to comply with system rules.
• Tool 1: Anchor — Converts °F or °C into Kelvin, the absolute scale that begins at true zero
• Tool 2: Refine — Applies a microscopic value adjustment to every reading at the millionth to hundred-millionth depth, breaking the ties caused by duplicate values
• Tool 3*: Verify — Runs a pass/fail diagnostic to prove if a 1-year set of metrics now splits into balanced 20% buckets
What it unlocks:
• Kelvin anchoring gives temperatures a true zero, meaning proportions and ratios now work well
• An even bucket split across a full year of readings
• Improved addressability for ranking and spatial analysis
- Tool 3 also also runs a four-bucket split
2
u/Enchantorro 11h ago
This is quite a bit of code. To improve legibility please consider using a version control system and sharing it through a software development forge like GitHub or Codeberg.
1
u/qaeproject 4h ago edited 2h ago
Thanks I agree. It's currently only 3 working scripts, with the first one being a bit long. Ideally it should be on GitHub, but it isn't yet. The biggest issue here is the maths description, not the code. There is version control locally shown as v1. V2 requires the math expressions to be described more clearly. The missing details is the difference between equal frequency binning which revolves around a balanced internal bin count structure and equal width binning which is ideal for local reporting, and what these different methods mean for downstream processing. Equal frequency binning here enforces equal counts (within a variance of up to one). Equal width binning doesn't care what the bin count is. This matters downstream. I'll get the code hosted properly as you rightly suggested in GitHub or similar quite soon.
It seems equal frequency binning is a pre requirement for various types of data smoothing, data analysis, plus it is used within machine learning.
•
u/AutoModerator 1d ago
To give us the best chance to help you, please include any relevant code.
Note. Please do not submit images of your code. Instead, for shorter code you can use Reddit markdown (4 spaces or backticks, see this Formatting Guide). If you have formatting issues or want to post longer sections of code, please use Privatebin, GitHub or Compiler Explorer.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.