<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>That's strange. Doesn't <font face="monospace">Arena::grow() </font>allocate
a chunk of size Chunk::size (~32K) or larger?</p>
<p>dl<br>
</p>
<div class="moz-cite-prefix">On 2/10/23 12:42 PM, Justin King wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAEdz=WWn=BsKAanNDuYVT0wB_v=FYw-2RicM+kOk501U+FhiHQ@mail.gmail.com">
<div dir="ltr">Hm. That size is still too big, I don't think the
node is 224 bytes. Let me double check again.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Feb 10, 2023 at 12:06
PM Justin King <<a href="mailto:jcking@google.com" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Looks to be the temporary node created by
clone_map is leaking. I think there is a missing call to
undo some of the work done by clone_map.
<div><br>
</div>
<div><font face="monospace">==3591618==ERROR:
AddressSanitizer: use-after-poison on address
0x6110000e20b8 at pc 0x7f7d8cac1875 bp 0x7f7d1028be20 sp
0x7f7d1028be18<br>
READ of size 4 at 0x6110000e20b8 thread T13<br>
#0 0x7f7d8cac1874 in
Unique_Node_List::remove_useless_nodes(VectorSet&)
src/hotspot/share/opto/node.cpp:2967<br>
#1 0x7f7d8cc90b03 in
PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*,
Unique_Node_List*, Phase::PhaseNumber)
src/hotspot/share/opto/phaseX.cpp:423<br>
#2 0x7f7d8b03853b in Compile::Compile(ciEnv*,
ciMethod*, int, Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:797<br>
#3 0x7f7d8ace8ece in
C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool,
DirectiveSet*) src/hotspot/share/opto/c2compiler.cpp:113<br>
#4 0x7f7d8b0507f8 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#5 0x7f7d8b053e57 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#6 0x7f7d8bc1f3a8 in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#7 0x7f7d8bc1f99f in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#8 0x7f7d8bc1f99f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#9 0x7f7d8d535b35 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#10 0x7f7d8cb7de8f in thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#11 0x7f7d8faa7fd3 in start_thread
nptl/pthread_create.c:442<br>
#12 0x7f7d8fb2866b in clone3
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
<br>
0x6110000e20b8 is located 56 bytes inside of 224-byte
region [0x6110000e2080,0x6110000e2160)<br>
allocated by thread T13 here:<br>
#0 0x7f7d8fcb89cf in __interceptor_malloc
../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
#1 0x7f7d8cb65b9a in os::malloc(unsigned long,
MEMFLAGS, NativeCallStack const&)
src/hotspot/share/runtime/os.cpp:672<br>
#2 0x7f7d8a68c47e in Chunk::operator new(unsigned
long, AllocFailStrategy::AllocFailEnum, unsigned long)
src/hotspot/share/memory/arena.cpp:190<br>
#3 0x7f7d8a68c47e in Arena::grow(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.cpp:345<br>
#4 0x7f7d8cac1c44 in
Arena::internal_amalloc(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:113<br>
#5 0x7f7d8cac1c44 in Arena::AmallocWords(unsigned
long, AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:140<br>
#6 0x7f7d8cac1c44 in Node::clone() const
src/hotspot/share/opto/node.cpp:495<br>
#7 0x7f7d8b980c38 in GraphKit::clone_map()
src/hotspot/share/opto/graphKit.cpp:727<br>
#8 0x7f7d8c538b44 in
LibraryCallKit::inline_unsafe_load_store(BasicType,
LibraryCallKit::LoadStoreKind,
LibraryCallKit::AccessKind)
src/hotspot/share/opto/library_call.cpp:2599<br>
#9 0x7f7d8c5aad3e in
LibraryCallKit::try_to_inline(int)
src/hotspot/share/opto/library_call.cpp:416<br>
#10 0x7f7d8c5ae34b in
LibraryIntrinsic::generate(JVMState*)
src/hotspot/share/opto/library_call.cpp:116<br>
#11 0x7f7d8b4430e9 in Parse::do_call()
src/hotspot/share/opto/doCall.cpp:662<br>
#12 0x7f7d8cc4c1ef in Parse::do_one_bytecode()
src/hotspot/share/opto/parse2.cpp:2704<br>
#13 0x7f7d8cc1f3f6 in Parse::do_one_block()
src/hotspot/share/opto/parse1.cpp:1554<br>
#14 0x7f7d8cc207ce in Parse::do_all_blocks()
src/hotspot/share/opto/parse1.cpp:706<br>
#15 0x7f7d8cc2a214 in Parse::Parse(JVMState*,
ciMethod*, float) src/hotspot/share/opto/parse1.cpp:613<br>
#16 0x7f7d8acec235 in
ParseGenerator::generate(JVMState*)
src/hotspot/share/opto/callGenerator.cpp:99<br>
#17 0x7f7d8b03738c in Compile::Compile(ciEnv*,
ciMethod*, int, Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:763<br>
#18 0x7f7d8ace8ece in
C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool,
DirectiveSet*) src/hotspot/share/opto/c2compiler.cpp:113<br>
#19 0x7f7d8b0507f8 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#20 0x7f7d8b053e57 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#21 0x7f7d8bc1f3a8 in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#22 0x7f7d8bc1f99f in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#23 0x7f7d8bc1f99f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#24 0x7f7d8d535b35 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#25 0x7f7d8cb7de8f in thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#26 0x7f7d8faa7fd3 in start_thread
nptl/pthread_create.c:442</font><br>
<br>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Feb 9, 2023 at
2:31 PM Justin King <<a href="mailto:jcking@google.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hm. It unfortunately does not show where it
was poisoned, as the Arena uses large chunks with
multiple separate allocations per chunk. ASan only keeps
track of malloc/free. But if I change the Arena
implementation when building under ASan to just use a
right-sized chunk for each request, it should be able to
show us. I'll try that and bump this once I get
something more definitive.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Feb 9, 2023 at
1:41 PM <<a href="mailto:dean.long@oracle.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">dean.long@oracle.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Can ASan show where the memory was freed? We've
had crashes in the past (like JDK-8270028) that
could be related to memory corruption or how
ResourceArea recycles memory. The allocation
below seems to be using an Arena without a
ResourceArea, but if some other code used the same
arena wrapped in a ResourceArea, then it seems
like that could lead to potential problems.<br>
</p>
<p>dl<br>
</p>
<div>On 2/9/23 8:59 AM, Justin King wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I was looking at instrumenting Arena again
for ASan. The WIP patch is <a href="https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253" target="_blank" moz-do-not-send="true">047d4aa9a091cf5a84b9308454862e39666ca253</a>.
I ran back into the <a href="https://bugs.openjdk.org/browse/JDK-8298984" target="_blank" moz-do-not-send="true">suspicious
logic in C2</a> where nodes are used after
calling Arena::Afree. The first issue is
present in Node::destruct, which I fixed by
moving the call to Arena::Afree to the bottom
of the function (addressed in patch). The
second issue came up after in
Compile::Compile, the stack trace is below. It
looks like there are residual freed nodes
being operated on? Maybe we are failing to
unregister a temporary node from the node
list? Maybe related to clone_map which returns
SafePointNode?</div>
<div><br>
</div>
<div><font face="monospace">==3146540==ERROR:
AddressSanitizer: use-after-poison on
address 0x62d00996b370 at pc 0x7f9f93048335
bp 0x7f9ed29fae20 sp 0x7f9ed29fae18<br>
READ of size 4 at 0x62d00996b370 thread T13<br>
#0 0x7f9f93048334 in
Unique_Node_List::remove_useless_nodes(VectorSet&)
src/hotspot/share/opto/node.cpp:2967<br>
#1 0x7f9f932124b3 in
PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*,
Unique_Node_List*, Phase::PhaseNumber)
src/hotspot/share/opto/phaseX.cpp:423<br>
#2 0x7f9f91621beb in
Compile::Compile(ciEnv*, ciMethod*, int,
Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:797<br>
#3 0x7f9f912e37fe in
C2Compiler::compile_method(ciEnv*,
ciMethod*, int, bool, DirectiveSet*)
src/hotspot/share/opto/c2compiler.cpp:113<br>
#4 0x7f9f91638e07 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#5 0x7f9f9163bfd7 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#6 0x7f9f921e3eec in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#7 0x7f9f921e434f in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#8 0x7f9f921e434f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#9 0x7f9f93aa3f55 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#10 0x7f9f9310144f in
thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#11 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
#12 0x7f9f9612866b in clone3
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
<br>
0x62d00996b370 is located 20336 bytes inside
of 32744-byte region
[0x62d009966400,0x62d00996e3e8)<br>
allocated by thread T13 here:<br>
#0 0x7f9f962b89cf in
__interceptor_malloc
../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
#1 0x7f9f930e915a in os::malloc(unsigned
long, MEMFLAGS, NativeCallStack const&)
src/hotspot/share/runtime/os.cpp:672<br>
#2 0x7f9f90c8a08a in Chunk::operator
new(unsigned long,
AllocFailStrategy::AllocFailEnum, unsigned
long) src/hotspot/share/memory/arena.cpp:190<br>
#3 0x7f9f90c8a08a in
Arena::grow(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.cpp:325<br>
#4 0x7f9f932109f5 in
Arena::internal_amalloc(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:113<br>
#5 0x7f9f932109f5 in
Arena::Amalloc(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:133<br>
#6 0x7f9f932109f5 in
NodeHash::NodeHash(Arena*, unsigned int)
src/hotspot/share/opto/phaseX.cpp:68<br>
#7 0x7f9f932293c7 in
PhaseValues::PhaseValues(Arena*, unsigned
int) src/hotspot/share/opto/phaseX.cpp:697<br>
#8 0x7f9f9161f678 in
PhaseGVN::PhaseGVN(Arena*, unsigned int)
src/hotspot/share/opto/phaseX.hpp:415<br>
#9 0x7f9f9161f678 in
Compile::Compile(ciEnv*, ciMethod*, int,
Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:714<br>
#10 0x7f9f912e37fe in
C2Compiler::compile_method(ciEnv*,
ciMethod*, int, bool, DirectiveSet*)
src/hotspot/share/opto/c2compiler.cpp:113<br>
#11 0x7f9f91638e07 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#12 0x7f9f9163bfd7 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#13 0x7f9f921e3eec in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#14 0x7f9f921e434f in
JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#15 0x7f9f921e434f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#16 0x7f9f93aa3f55 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#17 0x7f9f9310144f in
thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#18 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
<br>
Thread T13 created by T1 here:<br>
#0 0x7f9f96249726 in
__interceptor_pthread_create
../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
#1 0x7f9f93102d88 in
os::create_thread(Thread*, os::ThreadType,
unsigned long)
src/hotspot/os/linux/os_linux.cpp:888<br>
#2 0x7f9f91693d93 in
CompilerThread::CompilerThread(CompileQueue*,
CompilerCounters*)
src/hotspot/share/compiler/compilerThread.cpp:34<br>
#3 0x7f9f91625c7c in
CompileBroker::make_thread(CompileBroker::ThreadType,
_jobject*, CompileQueue*, AbstractCompiler*,
JavaThread*)
src/hotspot/share/compiler/compileBroker.cpp:842<br>
#4 0x7f9f91628f71 in
CompileBroker::init_compiler_threads()
src/hotspot/share/compiler/compileBroker.cpp:943<br>
#5 0x7f9f9162a464 in
CompileBroker::compilation_init_phase1(JavaThread*)
src/hotspot/share/compiler/compileBroker.cpp:654<br>
#6 0x7f9f93adc3a4 in
Threads::create_vm(JavaVMInitArgs*, bool*)
src/hotspot/share/runtime/threads.cpp:701<br>
#7 0x7f9f92465b51 in
JNI_CreateJavaVM_inner
src/hotspot/share/prims/jni.cpp:3588<br>
#8 0x7f9f92465b51 in JNI_CreateJavaVM
src/hotspot/share/prims/jni.cpp:3674<br>
#9 0x7f9f968d2e25 in InitializeJVM
src/java.base/share/native/libjli/java.c:1459<br>
#10 0x7f9f968d2e25 in JavaMain
src/java.base/share/native/libjli/java.c:413<br>
#11 0x7f9f968db708 in ThreadJavaMain
src/java.base/unix/native/libjli/java_md.c:650<br>
#12 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
<br>
Thread T1 created by T0 here:<br>
#0 0x7f9f96249726 in
__interceptor_pthread_create
../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
#1 0x7f9f968dd3a1 in
CallJavaMainInNewThread
src/java.base/unix/native/libjli/java_md.c:691<br>
#2 0x7f9f968d822d in ContinueInNewThread
src/java.base/share/native/libjli/java.c:2280<br>
#3 0x7f9f968d96ae in JLI_Launch
src/java.base/share/native/libjli/java.c:340<br>
#4 0x5594a81c337c in main
src/java.base/share/native/launcher/main.c:166<br>
#5 0x7f9f96046189 in
__libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58</font><br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr"><span><br>
<table style="margin:0px;padding:20px
0px 0px;font-family:"Times New
Roman"" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td style="padding:0px 20px 0px
0px;vertical-align:top;border-right:1px
solid rgb(213,213,213)" valign="top"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" style="margin: 0px; padding:
0px; display: block; height:
auto;" moz-do-not-send="true" width="72"></td>
<td style="padding:0px 0px 0px
20px">
<table style="margin:0px;padding:0px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:1px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin
King</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software
Engineer</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table style="border:none;border-collapse:collapse">
<colgroup><col width="85"><col width="539"></colgroup><tbody>
<tr style="height:0pt">
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr"><span><br>
<table style="margin:0px;padding:20px 0px
0px;font-family:"Times New Roman"" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td style="padding:0px 20px 0px
0px;vertical-align:top;border-right:1px
solid rgb(213,213,213)" valign="top"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" style="margin: 0px;
padding: 0px; display: block; height:
auto;" moz-do-not-send="true" width="72"></td>
<td style="padding:0px 0px 0px 20px">
<table style="margin:0px;padding:0px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:1px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin
King</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software
Engineer</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table style="border:none;border-collapse:collapse">
<colgroup><col width="85"><col width="539"></colgroup><tbody>
<tr style="height:0pt">
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr"><span><br>
<table style="margin:0px;padding:20px 0px
0px;font-family:"Times New Roman"" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td style="padding:0px 20px 0px
0px;vertical-align:top;border-right:1px solid
rgb(213,213,213)" valign="top"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" style="margin: 0px; padding:
0px; display: block; height: auto;" moz-do-not-send="true" width="72"></td>
<td style="padding:0px 0px 0px 20px">
<table style="margin:0px;padding:0px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:1px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin
King</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software
Engineer</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table style="border:none;border-collapse:collapse">
<colgroup><col width="85"><col width="539"></colgroup><tbody>
<tr style="height:0pt">
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr"><span><br>
<table style="margin:0px;padding:20px 0px
0px;font-family:"Times New Roman"" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td style="padding:0px 20px 0px
0px;vertical-align:top;border-right:1px solid
rgb(213,213,213)" valign="top"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" style="margin: 0px; padding:
0px; display: block; height: auto;" moz-do-not-send="true" width="72"></td>
<td style="padding:0px 0px 0px 20px">
<table style="margin:0px;padding:0px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:1px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin
King</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software
Engineer</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px
5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">jcking@google.com</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table style="border:none;border-collapse:collapse">
<colgroup><col width="85"><col width="539"></colgroup><tbody>
<tr style="height:0pt">
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</blockquote>
</body>
</html>