<div dir="ltr">Yeah, I think that is it. Anybody have any idea what the intention is supposed to be with PhaseIterGVN and it's copying by value of Unique_Node_List and Node_Stack? Both of its constructors do this, one happens to copy Compile::for_igvn() and ultimately modify it. Is it supposed to be taking ownership, are any changes by PhaseIterGVN to either supposed to be visible to the caller, or is it meant to be like copy-on-write where changes are not visible to the caller?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 16, 2023 at 7:18 AM Justin King <<a href="mailto:jcking@google.com">jcking@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Okay, continuing to look for the remaining surprises. I think I found it, but I need to verify. It looks like PhaseIterGVN is copying Compile::for_igvn() by value and modifying it either intentionally or unintentionally, triggering Arealloc. This results in Compile::for_igvn() now pointing to free'd memory. I'll see if I can verify.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 15, 2023 at 9:05 AM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I sent <a href="https://github.com/openjdk/jdk/pull/12577" target="_blank">https://github.com/openjdk/jdk/pull/12577</a> and <a href="https://github.com/openjdk/jdk/pull/12578" target="_blank">https://github.com/openjdk/jdk/pull/12578</a> with their corresponding created bugs. I'll continue looking for the others once those are in.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 14, 2023 at 4:02 PM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Looks like there is definitely a lingering use-after-free like bug, maybe multiple. One is related to the copying around of Unique_Node_List and the copy then modifying the list, causing a Arealloc, with the original (which is then used again) seeing the free'd block. I'll see about pinpointing the issues at a later date.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 14, 2023 at 9:27 AM Vladimir Kozlov <<a href="mailto:vladimir.kozlov@oracle.com" target="_blank">vladimir.kozlov@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Justin,<br>
<br>
Please file a bug for this issue. We need to keep records.<br>
<br>
Thanks,<br>
Vladimir K<br>
<br>
On 2/10/23 12:42 PM, Justin King wrote:<br>
> Hm. That size is still too big, I don't think the node is 224 bytes. Let me double check again.<br>
> <br>
> On Fri, Feb 10, 2023 at 12:06 PM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>>> wrote:<br>
> <br>
> Looks to be the temporary node created by clone_map is leaking. I think there is a missing call to undo some of the<br>
> work done by clone_map.<br>
> <br>
> ==3591618==ERROR: AddressSanitizer: use-after-poison on address 0x6110000e20b8 at pc 0x7f7d8cac1875 bp<br>
> 0x7f7d1028be20 sp 0x7f7d1028be18<br>
> READ of size 4 at 0x6110000e20b8 thread T13<br>
> #0 0x7f7d8cac1874 in Unique_Node_List::remove_useless_nodes(VectorSet&) src/hotspot/share/opto/node.cpp:2967<br>
> #1 0x7f7d8cc90b03 in PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*, Unique_Node_List*, Phase::PhaseNumber)<br>
> src/hotspot/share/opto/phaseX.cpp:423<br>
> #2 0x7f7d8b03853b in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
> src/hotspot/share/opto/compile.cpp:797<br>
> #3 0x7f7d8ace8ece in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
> src/hotspot/share/opto/c2compiler.cpp:113<br>
> #4 0x7f7d8b0507f8 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
> #5 0x7f7d8b053e57 in CompileBroker::compiler_thread_loop() src/hotspot/share/compiler/compileBroker.cpp:1916<br>
> #6 0x7f7d8bc1f3a8 in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
> #7 0x7f7d8bc1f99f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
> #8 0x7f7d8bc1f99f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
> #9 0x7f7d8d535b35 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
> #10 0x7f7d8cb7de8f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
> #11 0x7f7d8faa7fd3 in start_thread nptl/pthread_create.c:442<br>
> #12 0x7f7d8fb2866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
> <br>
> 0x6110000e20b8 is located 56 bytes inside of 224-byte region [0x6110000e2080,0x6110000e2160)<br>
> allocated by thread T13 here:<br>
> #0 0x7f7d8fcb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
> #1 0x7f7d8cb65b9a in os::malloc(unsigned long, MEMFLAGS, NativeCallStack const&)<br>
> src/hotspot/share/runtime/os.cpp:672<br>
> #2 0x7f7d8a68c47e in Chunk::operator new(unsigned long, AllocFailStrategy::AllocFailEnum, unsigned long)<br>
> src/hotspot/share/memory/arena.cpp:190<br>
> #3 0x7f7d8a68c47e in Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.cpp:345<br>
> #4 0x7f7d8cac1c44 in Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.hpp:113<br>
> #5 0x7f7d8cac1c44 in Arena::AmallocWords(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.hpp:140<br>
> #6 0x7f7d8cac1c44 in Node::clone() const src/hotspot/share/opto/node.cpp:495<br>
> #7 0x7f7d8b980c38 in GraphKit::clone_map() src/hotspot/share/opto/graphKit.cpp:727<br>
> #8 0x7f7d8c538b44 in LibraryCallKit::inline_unsafe_load_store(BasicType, LibraryCallKit::LoadStoreKind,<br>
> LibraryCallKit::AccessKind) src/hotspot/share/opto/library_call.cpp:2599<br>
> #9 0x7f7d8c5aad3e in LibraryCallKit::try_to_inline(int) src/hotspot/share/opto/library_call.cpp:416<br>
> #10 0x7f7d8c5ae34b in LibraryIntrinsic::generate(JVMState*) src/hotspot/share/opto/library_call.cpp:116<br>
> #11 0x7f7d8b4430e9 in Parse::do_call() src/hotspot/share/opto/doCall.cpp:662<br>
> #12 0x7f7d8cc4c1ef in Parse::do_one_bytecode() src/hotspot/share/opto/parse2.cpp:2704<br>
> #13 0x7f7d8cc1f3f6 in Parse::do_one_block() src/hotspot/share/opto/parse1.cpp:1554<br>
> #14 0x7f7d8cc207ce in Parse::do_all_blocks() src/hotspot/share/opto/parse1.cpp:706<br>
> #15 0x7f7d8cc2a214 in Parse::Parse(JVMState*, ciMethod*, float) src/hotspot/share/opto/parse1.cpp:613<br>
> #16 0x7f7d8acec235 in ParseGenerator::generate(JVMState*) src/hotspot/share/opto/callGenerator.cpp:99<br>
> #17 0x7f7d8b03738c in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
> src/hotspot/share/opto/compile.cpp:763<br>
> #18 0x7f7d8ace8ece in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
> src/hotspot/share/opto/c2compiler.cpp:113<br>
> #19 0x7f7d8b0507f8 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
> #20 0x7f7d8b053e57 in CompileBroker::compiler_thread_loop() src/hotspot/share/compiler/compileBroker.cpp:1916<br>
> #21 0x7f7d8bc1f3a8 in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
> #22 0x7f7d8bc1f99f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
> #23 0x7f7d8bc1f99f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
> #24 0x7f7d8d535b35 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
> #25 0x7f7d8cb7de8f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
> #26 0x7f7d8faa7fd3 in start_thread nptl/pthread_create.c:442<br>
> <br>
> <br>
> <br>
> On Thu, Feb 9, 2023 at 2:31 PM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>>> wrote:<br>
> <br>
> Hm. It unfortunately does not show where it was poisoned, as the Arena uses large chunks with multiple separate<br>
> allocations per chunk. ASan only keeps track of malloc/free. But if I change the Arena implementation when<br>
> building under ASan to just use a right-sized chunk for each request, it should be able to show us. I'll try<br>
> that and bump this once I get something more definitive.<br>
> <br>
> On Thu, Feb 9, 2023 at 1:41 PM <<a href="mailto:dean.long@oracle.com" target="_blank">dean.long@oracle.com</a> <mailto:<a href="mailto:dean.long@oracle.com" target="_blank">dean.long@oracle.com</a>>> wrote:<br>
> <br>
> Can ASan show where the memory was freed? We've had crashes in the past (like JDK-8270028) that could be<br>
> related to memory corruption or how ResourceArea recycles memory. The allocation below seems to be using an<br>
> Arena without a ResourceArea, but if some other code used the same arena wrapped in a ResourceArea, then it<br>
> seems like that could lead to potential problems.<br>
> <br>
> dl<br>
> <br>
> On 2/9/23 8:59 AM, Justin King wrote:<br>
>> Hi,<br>
>><br>
>> I was looking at instrumenting Arena again for ASan. The WIP patch is<br>
>> 047d4aa9a091cf5a84b9308454862e39666ca253<br>
>> <<a href="https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253" rel="noreferrer" target="_blank">https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253</a>>. I ran back into the<br>
>> suspicious logic in C2 <<a href="https://bugs.openjdk.org/browse/JDK-8298984" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8298984</a>> where nodes are used after calling<br>
>> Arena::Afree. The first issue is present in Node::destruct, which I fixed by moving the call to<br>
>> Arena::Afree to the bottom of the function (addressed in patch). The second issue came up after in<br>
>> Compile::Compile, the stack trace is below. It looks like there are residual freed nodes being operated<br>
>> on? Maybe we are failing to unregister a temporary node from the node list? Maybe related to clone_map<br>
>> which returns SafePointNode?<br>
>><br>
>> ==3146540==ERROR: AddressSanitizer: use-after-poison on address 0x62d00996b370 at pc 0x7f9f93048335 bp<br>
>> 0x7f9ed29fae20 sp 0x7f9ed29fae18<br>
>> READ of size 4 at 0x62d00996b370 thread T13<br>
>> #0 0x7f9f93048334 in Unique_Node_List::remove_useless_nodes(VectorSet&)<br>
>> src/hotspot/share/opto/node.cpp:2967<br>
>> #1 0x7f9f932124b3 in PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*, Unique_Node_List*,<br>
>> Phase::PhaseNumber) src/hotspot/share/opto/phaseX.cpp:423<br>
>> #2 0x7f9f91621beb in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
>> src/hotspot/share/opto/compile.cpp:797<br>
>> #3 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
>> src/hotspot/share/opto/c2compiler.cpp:113<br>
>> #4 0x7f9f91638e07 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
>> #5 0x7f9f9163bfd7 in CompileBroker::compiler_thread_loop()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:1916<br>
>> #6 0x7f9f921e3eec in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
>> #7 0x7f9f921e434f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
>> #8 0x7f9f921e434f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
>> #9 0x7f9f93aa3f55 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
>> #10 0x7f9f9310144f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
>> #11 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>> #12 0x7f9f9612866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
>><br>
>> 0x62d00996b370 is located 20336 bytes inside of 32744-byte region [0x62d009966400,0x62d00996e3e8)<br>
>> allocated by thread T13 here:<br>
>> #0 0x7f9f962b89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
>> #1 0x7f9f930e915a in os::malloc(unsigned long, MEMFLAGS, NativeCallStack const&)<br>
>> src/hotspot/share/runtime/os.cpp:672<br>
>> #2 0x7f9f90c8a08a in Chunk::operator new(unsigned long, AllocFailStrategy::AllocFailEnum, unsigned<br>
>> long) src/hotspot/share/memory/arena.cpp:190<br>
>> #3 0x7f9f90c8a08a in Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.cpp:325<br>
>> #4 0x7f9f932109f5 in Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.hpp:113<br>
>> #5 0x7f9f932109f5 in Arena::Amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.hpp:133<br>
>> #6 0x7f9f932109f5 in NodeHash::NodeHash(Arena*, unsigned int) src/hotspot/share/opto/phaseX.cpp:68<br>
>> #7 0x7f9f932293c7 in PhaseValues::PhaseValues(Arena*, unsigned int) src/hotspot/share/opto/phaseX.cpp:697<br>
>> #8 0x7f9f9161f678 in PhaseGVN::PhaseGVN(Arena*, unsigned int) src/hotspot/share/opto/phaseX.hpp:415<br>
>> #9 0x7f9f9161f678 in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
>> src/hotspot/share/opto/compile.cpp:714<br>
>> #10 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
>> src/hotspot/share/opto/c2compiler.cpp:113<br>
>> #11 0x7f9f91638e07 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
>> #12 0x7f9f9163bfd7 in CompileBroker::compiler_thread_loop()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:1916<br>
>> #13 0x7f9f921e3eec in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
>> #14 0x7f9f921e434f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
>> #15 0x7f9f921e434f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
>> #16 0x7f9f93aa3f55 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
>> #17 0x7f9f9310144f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
>> #18 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>><br>
>> Thread T13 created by T1 here:<br>
>> #0 0x7f9f96249726 in __interceptor_pthread_create<br>
>> ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
>> #1 0x7f9f93102d88 in os::create_thread(Thread*, os::ThreadType, unsigned long)<br>
>> src/hotspot/os/linux/os_linux.cpp:888<br>
>> #2 0x7f9f91693d93 in CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*)<br>
>> src/hotspot/share/compiler/compilerThread.cpp:34<br>
>> #3 0x7f9f91625c7c in CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*,<br>
>> AbstractCompiler*, JavaThread*) src/hotspot/share/compiler/compileBroker.cpp:842<br>
>> #4 0x7f9f91628f71 in CompileBroker::init_compiler_threads()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:943<br>
>> #5 0x7f9f9162a464 in CompileBroker::compilation_init_phase1(JavaThread*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:654<br>
>> #6 0x7f9f93adc3a4 in Threads::create_vm(JavaVMInitArgs*, bool*) src/hotspot/share/runtime/threads.cpp:701<br>
>> #7 0x7f9f92465b51 in JNI_CreateJavaVM_inner src/hotspot/share/prims/jni.cpp:3588<br>
>> #8 0x7f9f92465b51 in JNI_CreateJavaVM src/hotspot/share/prims/jni.cpp:3674<br>
>> #9 0x7f9f968d2e25 in InitializeJVM src/java.base/share/native/libjli/java.c:1459<br>
>> #10 0x7f9f968d2e25 in JavaMain src/java.base/share/native/libjli/java.c:413<br>
>> #11 0x7f9f968db708 in ThreadJavaMain src/java.base/unix/native/libjli/java_md.c:650<br>
>> #12 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>><br>
>> Thread T1 created by T0 here:<br>
>> #0 0x7f9f96249726 in __interceptor_pthread_create<br>
>> ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
>> #1 0x7f9f968dd3a1 in CallJavaMainInNewThread src/java.base/unix/native/libjli/java_md.c:691<br>
>> #2 0x7f9f968d822d in ContinueInNewThread src/java.base/share/native/libjli/java.c:2280<br>
>> #3 0x7f9f968d96ae in JLI_Launch src/java.base/share/native/libjli/java.c:340<br>
>> #4 0x5594a81c337c in main src/java.base/share/native/launcher/main.c:166<br>
>> #5 0x7f9f96046189 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58<br>
>><br>
>> -- <br>
>><br>
>> Google Logo <br>
>> Justin King<br>
>> Software Engineer<br>
>> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
>><br>
>><br>
>> <br>
>><br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>